<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Louis Henri Franc</title>
    <description></description>
    <link>http://louishenrifranc.github.io/</link>
    <atom:link href="http://louishenrifranc.github.io/feed.xml" rel="self" type="application/rss+xml" />
    <pubDate>Sat, 23 Dec 2017 05:14:44 -0500</pubDate>
    <lastBuildDate>Sat, 23 Dec 2017 05:14:44 -0500</lastBuildDate>
    <generator>Jekyll v3.6.2</generator>
    
      <item>
        <title>Alpha Go Zero in depth</title>
        <description>&lt;h1 id=&quot;introduction&quot;&gt;Introduction&lt;/h1&gt;
&lt;p&gt;A long-standing objective of artificial intelligence is an algorithm that learns superhuman competence by itself. If man succeeds in constructing such a system, then the repercussions of this discovery will be numerous and diverse and will change our daily life as human beings. However, it is not clear how we could give machines the ability to reason and understand the world around them. As of today, many systems, such as Amazon Alexa, or Google Home act as substitutes, allowing us to blossom with the limits of artificial intelligence, the fruit of an ungrateful training on a huge quantity of collected data. 
These advanced systems extract complex patterns hidden in the data in order to reproduce the human behavior. This is the case in the generation of speech, the understanding of images, or even the understanding of human desires. We are still a long way from an intelligent machine that understands us, and that would be the equivalent of a human being.&lt;/p&gt;

&lt;p&gt;Earlier this month, DeepMind proposed a new algorithm capable of playing the game of &lt;a href=&quot;https://en.wikipedia.org/wiki/Go_ (games)&quot;&gt;Go&lt;/a&gt; without any assistance or human supervision. Despite the remarkable performance, it should be noted that this algorithm is based on the fact that the Go game is very well defined, providing a standard score for each move. For a machine, this corresponds to an environment that can be simulated quickly on a computer. Unfortunately, this is not the case for many complex human problems that are not yet well defined, nor modelable, and that above all require a certain knowledge and intelligence to be solved.&lt;/p&gt;

&lt;p&gt;In the first section, we will review the origin of Reinforcement Learning (RL), as well as some of the algorithms used in Alpha Go
In the second section, we will cover the two versions of Alpha Go, and their differences.
Finally, in the last part, we will see potential applications to this new algorithm&lt;/p&gt;

&lt;h1 id=&quot;on-the-origin-of-reinforcement-learning&quot;&gt;On the origin of reinforcement learning&lt;/h1&gt;

&lt;p&gt;Reinforcement learning is a field of research where we teach &lt;strong&gt;agents&lt;/strong&gt; to interact with an &lt;strong&gt;environment&lt;/strong&gt; . This interaction required the agent to choose actions \(A\), that move the agent between states \(S\). Associated with each state is a notion of reward \(r\) that accumulate over the previous actions. The goal of the agent is to maximize the reward in the long term, instead of the immediate reward.
When human creates the environment, they need to define how reward will be distributed. The agent needs to learn a policy \(\pi(action|state)\) that provide the next action to choose from a given state.
When defining the rewards, there is a clear distinction to make with other machine learning technique: Rewards should not be given as a way to teach the human how to reach the ultimate goal, but instead, uniquely if the agent arrives at the goal. The agent will have to learn on its own what is the best way to succeed. For example, if you teach a robot to move from place \(A\) to place \(B\), you should never assign a reward to a certain path chosen.&lt;/p&gt;

&lt;p&gt;Given what we have said in the previous paragraph, we want the agent to maximize the expected gain, i.e \(G_t = R_{t+1} + R_{t+2} + … + R_{t+T}\). But in case there is no terminal state, what we want instead is to maximize the &lt;strong&gt;discounted reward&lt;/strong&gt; \(\sum_{k=0} ^ {\inf} \lambda^k R_{t+k+1}\). If we unified the last two equation, we find that \(G_t = \sum_{k=0} ^ {T-t-1} \lambda^k R_{t+k+1}\).&lt;/p&gt;

&lt;p&gt;Reinforcement learning relies on the &lt;a href=&quot;https://en.wikipedia.org/wiki/Markov_property&quot;&gt;Markov assumption&lt;/a&gt;. As the policy \(\pi\) is defined as a probability distribution, we can apply the Markov property on it:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;The probability of moving to a new state \(s^{‘}\) is only conditioned on the previous state and the action taken.&lt;/li&gt;
  &lt;li&gt;The expected reward after moving from \(s\) to \(s^{‘}\) with action \(a\) is only determined by the last three “objects”.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Although this may seem restrictive at first glance, the trick lies in modeling the problem, so that it will always respect the Markov property. For example, if stated in the problem, the last two states always condition the next one, then you have to go back to the modelisation, and make sure that these last two states are represented as a single one.&lt;/p&gt;

&lt;p&gt;When the agent interacts with the environment, knowing and learning if it is good to be in this state, or if it is good to choose this action, is essential. For example, if we go back to our robot looking for a reference point, if to go to this point the robot can meet a river, it would be good that he learns to mark this place as “to avoid”.
It leads us to see two functions:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;The value function: \(V_{\pi}(state) = \mathbb{E}(G_t|S_t=state)\) which gives us the expected reward to be in state \(state\)&lt;/li&gt;
  &lt;li&gt;The state function: \(Q_{\pi}(state, action) = \mathbb{E}(G_t|S_t=state, A_t=action)\) which gives us the expected reward to be take the action \(action\) at state \(state\).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the agent maintains an average for each state, or for each tuple (action, state), we know that if \(t\) goes to infinity, we would learn the true values. Sadly, expectation is for the theory, and in practise we work with approximate expectation using a sample of experimentations:
&lt;script type=&quot;math/tex&quot;&gt;v_{\pi}(state) = \sum_{action} \pi(action|state) \sum_{state^{'}} p(state^{'}|action, state)[r(state^{'}, state, action) + \lambda v_{\pi}(state^{'})]&lt;/script&gt;.&lt;/p&gt;

&lt;p&gt;This formula, also known as the Bellman equation, is the foundation stone of all reinforcement learning problems. Here is a schema of what’s happening in the agent’s environment:
&lt;img src=&quot;/img/2017-10-23-alphago/value_function.png&quot; alt=&quot;Figure 1&quot; /&gt;&lt;/p&gt;

&lt;p&gt;As previously mentioned, if the number of experiences goes on forever, we would learn the true values. That means we could order the policies:
&lt;script type=&quot;math/tex&quot;&gt;\pi ^{'}\text{ is better than } \pi \Leftrightarrow v_{\pi^{'}}(s) &gt;= v_{\pi}(s)  \forall s&lt;/script&gt;&lt;/p&gt;

&lt;p&gt;In simple cases, if the Markov decision process is &lt;a href=&quot;https://en.wikipedia.org/wiki/Markov_decision_decision_decision_decision_process#Definition&quot;&gt;finished&lt;/a&gt; which means that the number of states and actions is finite, then the Bellman equations can be solved using a system of \(N\) equations, with N unknowns ((\(v_{\pi}(state)\text{ }\forall state\)). This property also assumes that the dynamics of the environment are completely known, i. e. \(p, r\) are constants. 
As a result, we can easily develop a policy that would maximize the value return on the discount reward (maximize long-term and short-term compensation).
If we try to learn the state function \(Q\), it’s even easier because arg-max is hidden.&lt;/p&gt;

&lt;h3 id=&quot;dynamic-programming&quot;&gt;Dynamic programming&lt;/h3&gt;
&lt;p&gt;To solve the equations, we will use fast algorithms, or intermediate result storage techniques, such as dynamic programming used by some reinforcement learning algorithms. 
Some reminders and properties:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;The existence and uniqueness of the value function are guaranteed if \(\lambda\) is less than 1.&lt;/li&gt;
  &lt;li&gt;If the system dynamics are known, the problem can be solved as a system with as many equations and unknowns as states&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In our case, iterative methods are faster. Here is the new formula of the calculated value function iteratively:
&lt;script type=&quot;math/tex&quot;&gt;v_{\pi}^{k+1}(state) = \sum_{action} \pi(action|state) \sum_{state^{'}} p(state^{'}|action, state)[r(state'{'}, state, action) + \lambda v_{\pi}^{k}(state^{'})]&lt;/script&gt;.
It is proved that as k goes to infinity, we learn the true values of \(v\).&lt;/p&gt;

&lt;p&gt;Note that between two iterations, we can update efficiently the value function for all states using matrix multiplication  operations, and cache the previous table, and the new one.
As long as a value of a state is updated “significantly” (a hyperparameter must be defined), the algorithm continues.&lt;/p&gt;

&lt;p&gt;The Policy Improvement Theorem tells us that \(Q(a, \pi^{‘}) &amp;gt;= v_{pi}(s) \forall s, \pi^{‘} &amp;gt; \pi, v_{\pi^{‘}}(s) &amp;gt;= v_{\pi}(s)\)
This equation is very important has it links the value and state function, and is implicitly used in the results below.&lt;/p&gt;

&lt;p&gt;We will now review two iterative algorithms used in practice to learn value functions, and get the optimal policy:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a-html/node20.html&quot;&gt;Policy Iteration&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a-html/node19.html&quot;&gt;Value Iteration&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The only difference between the two is that the Policy Iteration evaluates a new policy by updating the value function as a function of the current policy. In other words, the current policy function is used to determine the probability of moving from one state to another (see formulas in the link). The algorithm then updates the policy and then re-starts the evaluation until the policy no longer changes in the second phase. 
On the other hand, Value Iteration algorithm updates the value following a greedy policy that always chooses the most promising action. The returned policy is only updated at the end.&lt;br /&gt;
Seen in this way, Value Iteration requires less computation than Policy Iteration.&lt;/p&gt;

&lt;h3 id=&quot;monte-carlo-methods&quot;&gt;Monte Carlo methods&lt;/h3&gt;
&lt;p&gt;Until then, we have lived in a wonderful world, where the dynamics of the system were known, such as the final reward but also the transition table. Unfortunately, this is often not the case, and we need to rely on other methods to teach the agent a policy. Since Go has an unambiguous objective system and is a fully observable game, we can still use the first methods.&lt;/p&gt;

&lt;p&gt;Perhaps in another blog, I would cover these methods, where bootstrap episodes of the agent in its environment are used to learn a policy. Looking for &lt;a href=&quot;https://en.wikipedia.org/wiki/Temporal_difference_learning&quot;&gt;Time Difference Learning&lt;/a&gt; that merges Monte Carlo updates and dynamic programming, or &lt;a href=&quot;https://en.wikipedia.org/wiki/State%E2%E2%E2%80%93action%E2%80%93action%E2%80%93reward%E2%80%93action&quot;&gt;Sarsa&lt;/a&gt; is a good rabbit hole journey start.&lt;/p&gt;

&lt;h3 id=&quot;recent-progress-in-reinforcement-learning-with-neural-network-as-approximation-functions&quot;&gt;Recent progress in Reinforcement Learning with Neural Network as approximation functions&lt;/h3&gt;
&lt;p&gt;All games of perfect information have an optimal value function that can be determined using the reward at the end of the game. To solve this games, people have used search trees to compute recursively the optimal value function. For a game like Go, the possible sequence of moves possible grows exponentially as a function of the number of moves left. For Go, there are approximately 250 possible moves at every timestep, and 150 moves before the game finished. So the number of possibilities to evaluate is \(250^{150}\).
To reduce the computation bottleneck, the depth and the width of the tree can be diminished using heuristics. For example, depth can be reduced by replacing some subtrees with an approximated value representing the expected reward under this subtree. For the width, sampling only actions from a golden policy allows only searching promising subtree.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Monte Carlo Tree Search&lt;/strong&gt;&lt;sup id=&quot;fnref:monte_carlo&quot;&gt;&lt;a href=&quot;#fn:monte_carlo&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; is a method to estimate the value of any state in a search tree using Monte Carlo rollouts. Monte Carlo rollout is a technique to traverse top to bottom a tree following a quick policy. 
One might try to pre-compute all the value function’s values, but this is no small matter, as it would require way more than all storage capacity in the world.&lt;/p&gt;

&lt;p&gt;This is where &lt;a href=&quot;https://en.wikipedia.org/wiki/Artificial_neural_neural_neural_network&quot;&gt;neural networks&lt;/a&gt; comes in handy. Their importance lies in their ability to represent complex data, such as images, in an abstract way and to use them for predictive purposes, such as the estimation of a probability distribution over a set of actions. 
Here is an example &lt;sup id=&quot;fnref:atari&quot;&gt;&lt;a href=&quot;#fn:atari&quot; class=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt; of a neural network architecture that learns to play the Atari game from the screen printing of the game. The neural network produces a policy on possible actions, i. e. left, right, up, down, etc.
&lt;img src=&quot;/img/2017-10-23-alphago/neural_network.PNG&quot; alt=&quot;Figure 2&quot; /&gt;&lt;/p&gt;

&lt;p&gt;It should be noted that recent applications of neural networks as a function approximator have generated unprecedented performance in many areas. And DeepMind has been a major contributor to this discovery.&lt;/p&gt;

&lt;h1 id=&quot;and-in-the-industry-which-uses-learning-reinforcement&quot;&gt;And in the industry, which uses learning reinforcement?&lt;/h1&gt;
&lt;p&gt;In a recent presentation, Andrew Ng, ranked the value of current methods of machine learning, pointing out that most of the value created comes from available supervised methods, followed by unsupervised techniques, and reinforcement learning techniques.&lt;/p&gt;

&lt;p&gt;Reinforcement learning is not meant to play the last of classes until the end of the world, and recent breakthroughs in video games, open the way to potential applications in medicine, language understanding, and self-driving cars.&lt;/p&gt;

&lt;h1 id=&quot;alpha-go-deep-neural-networks-and-tree-searching&quot;&gt;Alpha Go, deep neural networks and tree-searching&lt;/h1&gt;

&lt;p&gt;In their first attempt to break the game of Go &lt;sup id=&quot;fnref:alpha_go&quot;&gt;&lt;a href=&quot;#fn:alpha_go&quot; class=&quot;footnote&quot;&gt;3&lt;/a&gt;&lt;/sup&gt;, Deepmind developed a brand new training pipeline allowing a machine to play Go, with an honorary title of 9 dan. The training pipeline is decoupled in different stages, where new modules are added to the machine, allowing it to develop its game.&lt;/p&gt;

&lt;h3 id=&quot;supervised-training-of-the-policy-network&quot;&gt;Supervised Training of the Policy Network&lt;/h3&gt;
&lt;p&gt;In the first part of the training, a neural network is trained to maximize the likelihood of picking the human move a, given any state. This training is completely supervised, using millions of positions from an open source dataset of master games, and the model is trained via gradient descent.
This training produces two models: the \(SL\), used a the main policy network and a smaller one, less accurate but quicker, called \(FR\).&lt;/p&gt;

&lt;h3 id=&quot;reinforcement-learning-of-the-policy-network&quot;&gt;Reinforcement Learning of the Policy Network&lt;/h3&gt;
&lt;p&gt;Only when the supervised model has a low Mean Squared Error on the prediction of human movement, will it moves on to the next training step.&lt;/p&gt;

&lt;p&gt;At this stage, the model is trained against a different version of itself. Opponents are randomly sampled from a pool of variants of SL. Over time, only the best models continue to play amongst themselves. 
All these models train and learn from their victories, i.e. defeats. The technique used is a variant of &lt;a href=&quot;http://www.scholarpedia.org/article/Policy_gradient_methods&quot;&gt;Policy Gradient methods&lt;/a&gt;. A reward function provides different signals to the model if the state is terminal, winning or losing. This signal is used in the gradient descent algorithm to optimize the parametrized policy network. If you want to know more, I suggest you take a look at this blog&lt;sup id=&quot;fnref:karphaty&quot;&gt;&lt;a href=&quot;#fn:karphaty&quot; class=&quot;footnote&quot;&gt;4&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;This new policy, called \(RL\), wins 80% against \(SL\).&lt;/p&gt;

&lt;h3 id=&quot;learning-to-evaluate-the-value-network&quot;&gt;Learning to evaluate the value network&lt;/h3&gt;
&lt;p&gt;The final step is the evaluation of board positions. In short, a model is trained to provide a score to each possible state in the game of Go. To do this, a neural network similar to \(SL\) learns to predict a single scalar value, instead of a policy, which is the state value. Training is supervised, as the sampled set consists of millions of distinct positions of parties played by different \(RLs\) against themselves. 
The value that the network must predict correctly is the final reward at any state in the game.&lt;/p&gt;

&lt;h3 id=&quot;improving-performance-by-combining-policy-and-value-into-an-mcts&quot;&gt;Improving performance by combining policy and value into an MCTS&lt;/h3&gt;
&lt;p&gt;At this stage of training, Alpha Go is composed of:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;A module that provides a policy over next actions,&lt;/li&gt;
  &lt;li&gt;A similar module, quicker and simpler.&lt;/li&gt;
  &lt;li&gt;A module that can evaluate the quality of a board position for a given player&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At prediction time, a Monte Carlo Search Tree is built to evaluate the best move starting from the current state as a root. The tree has nodes as game state and edges as (state, action) tuple forming links between the game states. The child node is always the state at the next step. This architecture looks like the tree in &lt;em&gt;Figure 1&lt;/em&gt;. An edge keeps tracks of multiple values: the joint probability \(p(state, action)\), the number of times the search traverses the edge, and the total and average action value.&lt;/p&gt;

&lt;p&gt;Like all MCTSs, the tree is traversed in 4 successively repeated steps:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;The tree selects the edges to pass through using a randomly initialized Q action value first&lt;/li&gt;
  &lt;li&gt;If a new node is discovered, the next action is selected using the policy predicted by \(SL\) for that state.&lt;/li&gt;
  &lt;li&gt;Each node is evaluated by the value network, and also by a Monte Carlo rollout with \(FR\) as a quick policy.&lt;/li&gt;
  &lt;li&gt;Finally, the Q action values are updated from bottom to top.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is what hid behind the algorithm that defeated Lee Sedol last year in a competition followed by all Go fans, and geeks from around the world. Even though the performance stunned the whole world, it took no more than a year to see the new version, Alpha Go zero,  capable of beating the previous version, Alpha Go Lee, 100 games to 0.&lt;/p&gt;

&lt;h1 id=&quot;alpha-go-zero-you-dont-change-a-winning-team-you-combine-it&quot;&gt;Alpha Go Zero&lt;sup id=&quot;fnref:alpha_go_zero&quot;&gt;&lt;a href=&quot;#fn:alpha_go_zero&quot; class=&quot;footnote&quot;&gt;5&lt;/a&gt;&lt;/sup&gt;, you don’t change a winning team, you combine it!&lt;/h1&gt;
&lt;p&gt;The previous model, although very clever, demanded:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;Human Knowledge&lt;/li&gt;
  &lt;li&gt;Several neural networks, two for policy function and one for position evaluation&lt;/li&gt;
  &lt;li&gt;Requires dozens of GPUs to play&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The new model simply requires:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;Trained from scratch without human expertise&lt;/li&gt;
  &lt;li&gt;A single neural network&lt;/li&gt;
  &lt;li&gt;Just 4 &lt;a href=&quot;https://en.wikipedia.org/wiki/Tensor_processing_unit&quot;&gt;TPUs&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The genius of Alpha Go Zero lies in the algorithm. This requires only one neural network, trained to provide a policy and a state value simultaneously.&lt;/p&gt;

&lt;p&gt;For each new gameplay, the neural network is training to predict a probability for each possible movement and also the expected return value, win or loose. To correct its errors, the network is given additional time for reasoning, where it can take advantage of an MCTS to determine a better state value and strategy. The network corrects its predictions based on the values found during MCTS.&lt;/p&gt;

&lt;p&gt;The procedure in MCTS is almost identical as the one in Alpha Go 2016:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;Build the tree from the current state of the game&lt;/li&gt;
  &lt;li&gt;Go down until meeting a leaf node. On the way down, the edges are chosen using the Q value affected to the edges. A leaf node is either a terminal node or a node with a \(depth ~= 1500\)&lt;/li&gt;
  &lt;li&gt;Compute the value function in this leaf state using the current neural network. If the game is finished, provide a synthetic evaluation of +1, -1.&lt;/li&gt;
  &lt;li&gt;Backtrack this information up until the root and update the \(Q value\). Given that we chose an action \(action\), the \(Q value\) becomes the value function of the child state traversed divided by the number of times this state-&amp;gt;action has been travelled.&lt;/li&gt;
  &lt;li&gt;Repeat.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To save computation, MCTS is not re-initialized every time because the next move will use one of its subtree where the root node will be the current state.&lt;/p&gt;

&lt;p&gt;The multi-objective loss has two terms:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;The squared error between the expected reward and the MCTS reward&lt;/li&gt;
  &lt;li&gt;The cross-entropy between the neural network predicted policy and the MCTS policy&lt;/li&gt;
  &lt;li&gt;(and an L2 regularization)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The dual objective regularizes the network to a common representation that supports multiple edge case.&lt;/p&gt;

&lt;p&gt;When Alpha Go learns to play, the next move is sampled using a softmax over the MCTS predicted probabilities, controlled by a temperature term that balanced the trade-off between evaluation and exploration.&lt;/p&gt;

&lt;p&gt;Finally, as training goes on, neural network weights are saved, and performance is compared between the most recent model and the previous models. Out of 100 games, the one that wins by a certain margin most of the games becomes the best player for the next iteration of self-training.
During this generation phase, the player moves from a position to another, following an almost greedy policy driven by an MCTS search, where the action taken is the one that got visited mostly during the tree search.&lt;/p&gt;

&lt;h3 id=&quot;why-does-it-work-so-well&quot;&gt;Why does it work so well?&lt;/h3&gt;
&lt;p&gt;The structure of the networks used in AlphaGo Zero and Master have some great intuition that should hopefuly give some indication as to why a seemingly simpler system is capable of beating its more complex predecessor (AlphaGo Lee) however one would be forgiven for the confusion as to why AlphaGo Zero is able to beat Master so convincingly. The underlying principle that likely gives the systems increased capabilities is the sharing of features between the policy and value networks that allows the learning of much nicer features. However there is something else going on to explain why AlphaGo Zero is able to beat AlphaGo Master (which also shares this same idea) 100 games to 0. The difference between Master and Zero is that Master shares the training pipeline with AlphaGo Lee in that it is first trained on human games, it also uses some hand crafted features from the earlier versions. AlphaGo Zero has none of this and is trained entirely using reenforcement learning. At first glance, one might assume that this would be less effective due to a less informative training signal but that is not the case, but why? This next section is purely speculation, but I expect it actually has to do with a kind of local minima that has been found by human gameplay such that no small permutations to the human learned technique is able to greatly increase its capabilities.&lt;/p&gt;

&lt;h1 id=&quot;so-why-should-we-care-anyway&quot;&gt;So why should we care anyway?&lt;/h1&gt;
&lt;p&gt;“the rules of Go are so elegant, organic and rigorously logical that if intelligent life forms exist elsewhere in the Universe, they almost certainly play Go”.- Edward Lasker&lt;/p&gt;

&lt;p&gt;Before alphago beat Lee Sedoul it was to be believed that a computer beating a world champion was at least a decade away, a game in which humans dedicate their lives to mastering was conquered by a program written by people who themselves would not stand a chance. Is this a defeat for go or a giant leap for reinforcement learning and by extension AI? AlphaGo zero has been able to achieve in a matter of days the knowledge that took Go masters and scholars thousands of years of collective intelligence to develop. It was also shown that many of the techniques that alphago developed for its self are identical to strategies that have been developed by go players over this time period. Alongside these however, alphago was able to develop its own strategies that have not before been seen by the world of go, this acomplishment alone is a truly tremendous result and one that DeepMind will forever have their flag marked upon. (like IBM do with chess)&lt;/p&gt;

&lt;p&gt;The nice biproduct of applying machine learning to games is that it is capable of hitting the wider media and allowing regular people to get a real world grasp of the state of research in AI and allow people to get excited about a field that so often receives only scare-mongering or overstated claims by the media.&lt;/p&gt;

&lt;p&gt;AlphaGo Zero offers a truly beautiful solution to the problems solved with the original alphago, the idea of shared representations is both logical and efficient. I would even go so far to say that it makes the already beautiful original alphago look clunky.&lt;/p&gt;

&lt;p&gt;I think one of the real contribution of this research is the demonstration that reinforcement learning upon earlier versions of its self is on its own capable of truly astonishing results, in fact much better than was achieved by learning from master games. This coupled with the novel approach of merging policy and value networks has resulted in the fantastic results that we see here. The hope is that this will guide us into a realm where reinforcement learning is not only feasable to be used as a final “fine tuning” as it so often is but a method that can be deployed from scratch to extremely complex problems, given only a simulator of the system we wish to master.&lt;/p&gt;

&lt;p&gt;My concern with this however is that the game of go offers us a faithful, accurate and importantly easy-to-compute simulator of the discrete and fully observable world of go. This scenario is one that we are not often forunate enough to have in the most valuable real world applications of reinforcement learning.&lt;/p&gt;

&lt;p&gt;This said, there are some exciting applications that &lt;em&gt;can&lt;/em&gt; be easily simulated such as control systems for autonomous vehicles or robots both of which are simulatable with a reasonable reliability. But these systems still leave the unanswered question for alphago as to whether it can generalise over the non-observability and stochasticity of these systems. One thing is for sure though, this work from Deepmind will be built on and utilised in all kinds of novel and exciting ways over the coming months and years and hopefully bring us closer to solving some of the most important problems faced by the world.&lt;/p&gt;

&lt;h1 id=&quot;acknowledgments&quot;&gt;Acknowledgments&lt;/h1&gt;
&lt;p&gt;A big thank you to &lt;a href=&quot;https://www.linkedin.com/in/ben-townsend-358138b5/&quot;&gt;Ben Townsend&lt;/a&gt; for his precious proofreading and sharing his point of view on the repercussions and applications of Alpha Go Zero.&lt;/p&gt;

&lt;h1 id=&quot;bibliography&quot;&gt;Bibliography&lt;/h1&gt;

&lt;div class=&quot;footnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:monte_carlo&quot;&gt;
      &lt;p&gt;Monte-Carlo Tree Search: A New Framework for Game AI from &lt;em&gt;Guillaume Chaslot, Sander Bakkes, Istvan Szita and Pieter Spronck&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:monte_carlo&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:atari&quot;&gt;
      &lt;p&gt;: Playing Atari with Deep Reinforcement Learning from &lt;em&gt;Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:atari&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:alpha_go&quot;&gt;
      &lt;p&gt;Mastering the Game of Go with Deep Neural Networks and Tree Search from &lt;em&gt;D Silver, A Huang, C J Maddison, A Guez, L Sifre, G van den Driessche, J Schrittwieser, I Antonoglou, V Panneershelvam, M Lanctot, S Dieleman, D Grewe, J Nham, N Kalchbrenner, I Sutskever, T Graepel, T Lillicrap, M Leach, K Kavukcuoglu, D Hassabis&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:alpha_go&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:karphaty&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;http://karpathy.github.io/2016/05/31/rl/&quot;&gt;Blog article&lt;/a&gt;&amp;nbsp;&lt;a href=&quot;#fnref:karphaty&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:alpha_go_zero&quot;&gt;
      &lt;p&gt;Mastering the game of Go without human knowledge from &lt;em&gt;David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel and Demis Hassabis&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:alpha_go_zero&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</description>
        <pubDate>Mon, 23 Oct 2017 21:39:00 -0400</pubDate>
        <link>http://louishenrifranc.github.io/mathblog/2017/10/24/alphago/</link>
        <guid isPermaLink="true">http://louishenrifranc.github.io/mathblog/2017/10/24/alphago/</guid>
        
        <category>math</category>
        
        <category>seq2seq</category>
        
        <category>machine-learning</category>
        
        <category>go</category>
        
        <category>reinforcement learning</category>
        
        
        <category>mathblog</category>
        
      </item>
    
      <item>
        <title>Sonnet and Attention is All You Need</title>
        <description>&lt;h1 id=&quot;introduction&quot;&gt;Introduction&lt;/h1&gt;

&lt;p&gt;In this article, I will show you why Sonnet is one of the greatest Tensorflow library, and why everyone should use it. To support my remarks, I’ll implement a new model from Google Brain team &lt;sup id=&quot;fnref:attenttion&quot;&gt;&lt;a href=&quot;#fn:attenttion&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;, &lt;em&gt;Transformer&lt;/em&gt;, which is trained to translate sentences without any recurrent neural network. This model is state of the art in translation task.&lt;/p&gt;

&lt;p&gt;At some point, during the article, I will make some back and forth to explane some detail about the model. Skip them if you want!&lt;/p&gt;

&lt;p&gt;As stated before, I will be using &lt;a href=&quot;https://github.com/deepmind/sonnet&quot;&gt;Sonnet&lt;/a&gt; all the way through the blog. It has been developed internally at DeepMind. In essence, the principal feature of Sonnet is to abstract away the scoping mechanism of Tensorflow. If your neural network needs to share a lot of weights, this feature is essential. While giving you powerful abstraction, all you have to do is to group graph operations as Module. If a module is used in multiple parts of the graph, the variables inside the module are reused (without having to specify &lt;code class=&quot;highlighter-rouge&quot;&gt;reuse_scope=True&lt;/code&gt;). With Sonnet, you write less code, but you also make it more modular and reusable. Finally, as a lot of modules are implemented as part of the core library, it is even more easy to write modular code in a Sonnet way. For example &lt;code class=&quot;highlighter-rouge&quot;&gt;snt.Linear&lt;/code&gt; is a fully connected layer, &lt;code class=&quot;highlighter-rouge&quot;&gt;snt.LSTM&lt;/code&gt; is an RNN Cell, &lt;code class=&quot;highlighter-rouge&quot;&gt;snt.BatchApply&lt;/code&gt; is a utility to apply a module to every tensor across one or multiple axes.&lt;/p&gt;

&lt;p&gt;When experimenting new models, you usually want to train your model on the training set, once in a while evaluate, compute some metrics, store the model weights, and restore from the last checkpoint. Tensorflow’s learn API is designed to make this job easy, letting you focus on developing your model. With Estimator, you only have to define two functions:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;input_fn&lt;/code&gt;: A generator that Estimator will use to retrieve data into feed to the model&lt;/li&gt;
  &lt;li&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;model_fn&lt;/code&gt;: A function that creates a graph, returns a loss function during training, predictions during inference.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;the-structure-of-the-repository&quot;&gt;The structure of the repository&lt;/h1&gt;
&lt;p&gt;The repository will be structured as followed:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;Estimator is created in &lt;code class=&quot;highlighter-rouge&quot;&gt;attention/algorithms/transformers&lt;/code&gt;.&lt;/li&gt;
  &lt;li&gt;All the core module are in &lt;code class=&quot;highlighter-rouge&quot;&gt;attention/modules/cores&lt;/code&gt; (Multi-Head Attention, Input Embedding, Add &amp;amp; Norm, Feed Forward).&lt;/li&gt;
  &lt;li&gt;Decoders and Encoders modules are respectively in &lt;code class=&quot;highlighter-rouge&quot;&gt;attention/modules/decoders&lt;/code&gt;, and in &lt;code class=&quot;highlighter-rouge&quot;&gt;attention/modules/encoders&lt;/code&gt;.&lt;/li&gt;
  &lt;li&gt;The full model is in &lt;code class=&quot;highlighter-rouge&quot;&gt;attention/models&lt;/code&gt;.&lt;/li&gt;
  &lt;li&gt;All training and data processing are run as micro services in &lt;code class=&quot;highlighter-rouge&quot;&gt;attention/services&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;implementation-of-the-transformer&quot;&gt;Implementation of the Transformer&lt;/h1&gt;
&lt;h2 id=&quot;the-encoder&quot;&gt;The Encoder&lt;/h2&gt;
&lt;p&gt;&lt;img style=&quot;float: right;&quot; src=&quot;/img/2017-08-15-attention/encoder.PNG&quot; width=&quot;200&quot; height=&quot;100&quot; /&gt;
The Encoder is a stack of identical layers (blocks). The Encoder receives as input a batch of padded sequence ids, with their respective length. The encoder output a batch of encoder output, where each sequences identifiers is now a non linearly transformed vector.&lt;/p&gt;

&lt;p&gt;Here is the code of the Encoder module:&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;Encoder&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;snt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;AbstractModule&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;params&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;block_params&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embed_params&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;nb&quot;&gt;super&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Encoder&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;encoder&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;params&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;params&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;block_params&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;block_params&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;embed_params&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embed_params&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;_build&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sequences_length&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;reuse_embeddings&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;True&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;

        &lt;span class=&quot;n&quot;&gt;positionnal_embedding&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PositionnalEmbedding&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;**&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;embed_params&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;positionnal_embedding&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

        &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;params&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout_rate&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;0.0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;layers&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;output&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;params&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout_rate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

        &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;_&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;range&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;params&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_blocks&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;encoder_block&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;EncoderBlock&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;**&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;block_params&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;encoder_block&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;output&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sequences_length&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;This module is one the high-level module that calls the other ones. The structure of this module consists of a constructor and a &lt;code class=&quot;highlighter-rouge&quot;&gt;_build()&lt;/code&gt; function. The object must inherit from &lt;code class=&quot;highlighter-rouge&quot;&gt;snt.AbstractModule&lt;/code&gt;`. A good practice is to pass in the constructor all parameters that will not vary between two calls of the module. In my case, I’m passing the parameters of the Encoder, of the EncoderBlock, and the Embedding Block. The last two will be passed directly to their own module’s constructor. Also, it is better not to instantiate Tensorflow variables in Sonnet module’s constructor.&lt;/p&gt;

&lt;p&gt;The next part is to create the graph of the module, connect inputs, and return outputs, all of that inside the &lt;code class=&quot;highlighter-rouge&quot;&gt;_build()&lt;/code&gt; function. You can call your module with the &lt;strong&gt;call&lt;/strong&gt;. The first time, it will call the _build and instantiate the variable in the Graph. Afterwards, if the module is called once more, the module will reuse variables created previously.&lt;/p&gt;

&lt;p&gt;For example, in the code above, when &lt;code class=&quot;highlighter-rouge&quot;&gt;EncoderBlock&lt;/code&gt; are stacked, I need to re-create a new object each time.&lt;/p&gt;

&lt;p&gt;And, here is the code for the EncoderBlock:&lt;/p&gt;
&lt;h2 id=&quot;the-encoder-block-module&quot;&gt;The Encoder block module:&lt;/h2&gt;
&lt;p&gt;&lt;img style=&quot;float: right;&quot; src=&quot;/img/2017-08-15-attention/encoderblock.PNG&quot; width=&quot;200&quot; height=&quot;100&quot; /&gt;&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;EncoderBlock&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;snt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;AbstractModule&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dropout_rate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;nb&quot;&gt;super&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;EncoderBlock&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;encoder_block&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt;
        &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout_rate&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dropout_rate&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;_build&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sequence_length&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;keys&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;queries&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;inputs&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;keys_len&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;  &lt;span class=&quot;n&quot;&gt;queries_len&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;sequence_length&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;MultiHeadAttention&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dropout_rate&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout_rate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
                                    &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;queries&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;queries&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;keys&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;keys&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; 
                                        &lt;span class=&quot;n&quot;&gt;queries_len&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;queries_len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;keys_len&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;keys_len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;queries&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;LayerNorm&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;output&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

        &lt;span class=&quot;n&quot;&gt;pointwise_module&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;PointWiseFeedForward&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hidden_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;output_size&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;output&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;get_shape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;as_list&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()[&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;dropout_rate&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dropout_rate&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;pointwise_module&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;output&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

        &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;LayerNorm&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;output&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;output&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h2 id=&quot;the-embedding-module&quot;&gt;The Embedding Module&lt;/h2&gt;
&lt;p&gt;&lt;img style=&quot;float: right;&quot; src=&quot;/img/2017-08-15-attention/embedding.PNG&quot; width=&quot;200&quot; height=&quot;100&quot; /&gt;&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;class&lt;/span&gt; &lt;span class=&quot;nc&quot;&gt;PositionnalEmbedding&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;snt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;AbstractModule&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;embed_dim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;nb&quot;&gt;super&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;PositionnalEmbedding&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;__init__&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;positional_embedding&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;with&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;_enter_variable_scope&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;():&lt;/span&gt;
            &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;embed&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;snt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Embed&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;
                &lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;vocab_size&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
                &lt;span class=&quot;n&quot;&gt;embed_dim&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;embed_dim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
                &lt;span class=&quot;n&quot;&gt;name&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;&quot;embedding&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;

    &lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;_build&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;ids&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;emb_lookup&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;embed&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;ids&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;positionnal_embedding&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;get_variable&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;'positional_embedding'&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dtype&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;float32&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;
                                                &lt;span class=&quot;n&quot;&gt;shape&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;emb_lookup&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;get_shape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;())&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;emb_lookup&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;positionnal_embedding&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;As you can see in the code, I created a variable in the constructor which is to avoid when possible. In my case, I don’t want to define the vocabulary size and embedding dimension as an attribute of the module, so I used them only to create the embedding dictionary in the constructor. The Tensorflow variable is created inside the Sonnet module’s scope with &lt;code class=&quot;highlighter-rouge&quot;&gt;enter_variable_scope&lt;/code&gt;. This convenient scoping function will not recreate variable defined inside its scope. But if you want to abstract the scoping mechanism of Tensorflow, just &lt;strong&gt;define everything in _build()&lt;/strong&gt;&lt;/p&gt;

&lt;h4 id=&quot;thoughts-on-the-paper-implementation&quot;&gt;Thoughts on the paper implementation&lt;/h4&gt;
&lt;p&gt;The author used fixed positional embedding to encode the position of a token in a sequence. These positional embeddings have the same shape as the token embeddings, and they are computed using cosine and sine of different frequency. In my implementation, I used learned positional embedding as it is simpler to implement and perform equivalently. One drawback is that prediction sequences can’t be larger than sequences encounter during training, as no positional embedding would exist.&lt;/p&gt;

&lt;p&gt;&lt;img style=&quot;float: right;&quot; src=&quot;/img/2017-08-15-attention/multihead.PNG&quot; width=&quot;200&quot; height=&quot;100&quot; /&gt;&lt;/p&gt;
&lt;h2 id=&quot;the-multi-head-attention-module&quot;&gt;The Multi Head Attention Module&lt;/h2&gt;
&lt;p&gt;The last module example will be the Multi Head Attention Module. If you are not interested in understanding the Transformer, you can skip the next part, otherwise, I’ll recommand you to read the next paragraphs.&lt;/p&gt;

&lt;h4 id=&quot;a-gentle-reminder-about-what-attention-is&quot;&gt;A gentle reminder about what attention is&lt;/h4&gt;
&lt;p&gt;Before seeing any code, let’s review together what attention is. Introduced by Badhanau &lt;sup id=&quot;fnref:bahdanau&quot;&gt;&lt;a href=&quot;#fn:bahdanau&quot; class=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;, it is a mechanism allowing to attend over a vector of values. These values are selected by combining a queries vector with a keys vector, each keys referring to a value.&lt;/p&gt;

&lt;p&gt;Imagine that you want to cook some chocolate cake, so you’ll need some chocolate, some butter, flour, and eggs. Ingredients will be the &lt;strong&gt;values&lt;/strong&gt;. For all these ingredients, you have your preference; and we all know you love chocolate. Keys will represent how much you enjoy the taste of these ingredients in the cake. Furthermore, each time you cook you always refer to your grandma’s recipe book. So you open it, search for the chocolate cake recipe, and look for each ingredient’s standard proportions (the &lt;strong&gt;Queries&lt;/strong&gt;). 
Now given each ingredient’s quantity, you want to have each component’s optimal ratio for your final recipe. If you love chocolate, it is likely you’ll add more of it than your grandma’s recipe. Hence, you will attend over your preference, to settle the initial quantities and get the final proportion of each ingredient.&lt;/p&gt;

&lt;p&gt;Back to our sheep, attention weights can be calculated as a dot product attention, i.e.,
&lt;script type=&quot;math/tex&quot;&gt;Attention(Queries, Keys, Values) = Softmax((Queries \cdot Keys^T)\cdot Values)&lt;/script&gt;&lt;/p&gt;

&lt;p&gt;In the module, there won’t be a single attention, but \(d_k\) ones. All queries, keys and values are vectors; they will be linearly transformed first, before being split into \(d_k\) smaller vectors. On each of these groups, we perform the multi head attention. Then, we collect all the weighted values and concatenate them back into a single vector of the same size as the queries.&lt;/p&gt;

&lt;p&gt;In practice, every layer performed multiple attentions.&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;In the encoder, as queries, values, and keys are the same, self-attention is made over the output of the previous layer.&lt;/li&gt;
  &lt;li&gt;On the decoder side, two form of attention are performed: one between the previous decoder output as queries, and the output of the encoder as keys and values; and one self-attention over the previous layer&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When we decode sequences during training, we do self-attention between the decoder input. To prevent the decoder from viewing future information, another mask is applied to the attention&lt;/p&gt;

&lt;p&gt;In practise, if \(Queries \cdot Queries^T\) is like that,&lt;/p&gt;
&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;keys

t=3    (0.4, 0.3, 0.1, 0.9)
t=2    (0.3, 1.2, 0.3, 0.6)
t=1    (0.1, 0.3, 1.9, 0.2) 
t=0    (0.9, 0.6, 0.2, 1.0)
     t=0  t=1  t=2  t=3    queries
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;we mask out some values before applying the Softmax.&lt;/p&gt;
&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;keys

t=3    (-inf, -inf, -inf, 0.9)
t=2    (-inf, -inf,  0.3, 0.6)
t=1    (-inf,  0.3,  1.9, 0.2) 
t=0    (0.9,   0.6,  0.2, 1.0)
     t=0    t=1   t=2  t=3    queries
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Note that we also apply a mask when sequences are initially padded.&lt;/p&gt;

&lt;h3 id=&quot;the-implementation&quot;&gt;The implementation&lt;/h3&gt;
&lt;p&gt;Now that I explained multi attention, let’s see in practice how it looks like:&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;_build&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;queries&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;keys&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;queries_len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;keys_len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;values&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;bp&quot;&gt;None&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;The module’s inputs are queries, keys, their respective lengths, and potentially values. In our implementation, we don’t need to pass values as they will always be equal to the keys.&lt;/p&gt;

&lt;p&gt;We embedded all keys, queries, and values into a same size’s space:&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;q_w&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;contrib&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;layers&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;fully_connected&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;queries&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;input_dim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# batch_size x query_l x d_model &lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;k_w&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;contrib&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;layers&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;fully_connected&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;keys&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;input_dim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# batch_size x keys_l x d_model &lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;v_w&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;contrib&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;layers&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;fully_connected&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;values&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;input_dim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# batch_size x values_l x d_model &lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;We then split them into &lt;code class=&quot;highlighter-rouge&quot;&gt;num_heads&lt;/code&gt; vectors. On each of this queries, keys and values, we apply attention in parallel&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c&quot;&gt;# batch_size x num_head x [queries|keys|values]_l x d_model / 8&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;q_wi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transpose&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;split&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;q_w&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;axis&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;k_wi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transpose&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;split&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;k_w&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;axis&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;v_wi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transpose&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;split&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;v_w&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;num_heads&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;axis&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;),&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;dot_product_att&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;query&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;head_i&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;matmul&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;query&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transpose&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;key&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]))&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;key&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;get_shape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;as_list&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()[&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;**&lt;/span&gt; &lt;span class=&quot;mf&quot;&gt;0.5&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;head_i&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;We multiply each queries with each keys.&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;dot_prod_op&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;snt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;BatchApply&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;dot_product_att&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;logits_q_wi_k_wi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;dot_prod_op&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;q_wi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;k_wi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# batch_size x num_heads x query_l x key_l&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;If multi attention is performed on the decoder output with keys equal the previous decoder’s output, we mask out as discussed earlier.&lt;/p&gt;

&lt;p&gt;&lt;img style=&quot;float: right;&quot; src=&quot;/img/2017-08-15-attention/multiheadmasked.PNG&quot; width=&quot;200&quot; height=&quot;100&quot; /&gt;&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;mask_leftward_decoder&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;logits_q_wi_k_wi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+=&lt;/span&gt; &lt;span class=&quot;bp&quot;&gt;self&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;create_mask_for_decoding&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;*&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;logits_q_wi_k_wi&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;get_shape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;as_list&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Afterward, we perform softmax&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;softmax_q_wi_k_wi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nn&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;softmax&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;logits_q_wi_k_wi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# batch_size x num_heads x queries_l x keys_l&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;to get a weighting of our values&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;attention_qwi_kwi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;matmul&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;softmax_q_wi_k_wi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;v_wi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;  &lt;span class=&quot;c&quot;&gt;# batch_size x num_heads x queries_l x d_model / 8&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;At the end, we concatenate again the output values, and project them linearly&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c&quot;&gt;# batch_size x queries_l x d_model / 8 x num_heads&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;attention_qwi_kwi&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;transpose&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;attention_qwi_kwi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;0&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;3&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;

&lt;span class=&quot;c&quot;&gt;# batch_size x queries_l x input_len&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;concat_attention&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;reshape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;attention_qwi_kwi&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;-&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;queries&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;get_shape&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;as_list&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()[&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;],&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;input_dim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;])&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;multi_attention&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;contrib&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;layers&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;fully_connected&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;concat_attention&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;input_dim&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;If you want to see the full code, go &lt;a href=&quot;https://github.com/louishenrifranc/attention/blob/master/attention/modules/core/multihead_attention.py&quot;&gt;there&lt;/a&gt;. Note that you can also define utility methods in a Sonnet modules, as long as you ensure not to call this method on their own, but inside the build function. Note that there are many ways to protect methods like the build one is (like &lt;code class=&quot;highlighter-rouge&quot;&gt;@snt.reuse_variables&lt;/code&gt;).&lt;/p&gt;

&lt;h2 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;In this article, I presented Sonnet. There is way more to know about this excellent library, and I would encourage everyone to have a look at their &lt;a href=&quot;https://deepmind.github.io/sonnet/api/&quot;&gt;documentation&lt;/a&gt;. 
If you have some thoughts about the article, or about Sonnet, please write them down in the comment section, I’ll be interested to discuss them. 
On a side note, I’m also very excited by the promising results of Transformer, and I will be happy to share with you my research on this … module …&lt;/p&gt;

&lt;h1 id=&quot;bibliography&quot;&gt;Bibliography&lt;/h1&gt;
&lt;div class=&quot;footnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:attenttion&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1706.03762&quot;&gt;Attention is all You Need&lt;/a&gt; from &lt;em&gt;Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:attenttion&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:bahdanau&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1409.0473&quot;&gt;Neural Machine Translation by Jointly Learning to Align and Translate&lt;/a&gt; from &lt;em&gt;Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio&lt;/em&gt;  will&amp;nbsp;&lt;a href=&quot;#fnref:bahdanau&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</description>
        <pubDate>Thu, 24 Aug 2017 21:39:00 -0400</pubDate>
        <link>http://louishenrifranc.github.io/techblog/2017/08/25/attention/</link>
        <guid isPermaLink="true">http://louishenrifranc.github.io/techblog/2017/08/25/attention/</guid>
        
        <category>tech</category>
        
        <category>tensorflow</category>
        
        <category>sonnet</category>
        
        <category>estimator</category>
        
        
        <category>techblog</category>
        
      </item>
    
      <item>
        <title>Injecting Stochastic Noise during Dialogue Generation.</title>
        <description>&lt;p&gt;In this blog, we’ll review a recent paper &lt;em&gt;A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues&lt;/em&gt; from Serban &amp;amp; al &lt;sup id=&quot;fnref:vhred&quot;&gt;&lt;a href=&quot;#fn:vhred&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;. This article introduces a new generative model called VHRED trained to produce meaningful conversation utterances using a latent variable \(z\) at the decoder. As the joint distribution of \(p(z, x)\) where \(x\) is our data is intractable, they appeal to resort to variational inference’s method. As I’m currently learning about variational inference, I thought it was a good time to explain the mathematical part of this paper.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Nota Bene&lt;/em&gt;: The following part does not require a knowledge of variational inference, neither it requires you to read VHRED paper, but if I had some extra time, I’ll start by reading this &lt;a href=&quot;https://www.cs.jhu.edu/~jason/tutorials/variational.html&quot;&gt;fabulous introduction to VI&lt;/a&gt;, and goes quickly over the paper :).&lt;/p&gt;

&lt;p&gt;This article is an extension of HRED, the hierarchical recurrent encoder-decoder &lt;sup id=&quot;fnref:hred&quot;&gt;&lt;a href=&quot;#fn:hred&quot; class=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;, which belongs to the family of sequence to sequence model. For dialogue generation, instead of generating an answer given a question, the model is conditioned on all the previous utterances in the conversation (question and answer). Each utterance is firstly encoded via a recurrent neural network RNN, which produces an output vector. Each of this sentences is then passed to a second RNN, which encodes all dialogue as a single vector. The decoder’s task is to generate a response conditioned on this vector.&lt;/p&gt;

&lt;p&gt;Model is trained to maximize the likelihood of the data via stocastic gradient descent. This likelihood can be modeled as a probability over the output of the recurrent neural network:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;p(x_1, x_2, x_N) = \prod_{i=1}^N p(x_1) p(x_i\| x_{i-1}... x_1)&lt;/script&gt;

&lt;p&gt;After having being trained, we can use our generative model to produce some utterances. To do so, we generate a new sentence, word after another, conditioning our word selection on all the previously sampled words. Usually, people use different techniques to sample a new word: Greedy sampling, Stochastic sampling. These methods only modify the output distribution of the model. VHRED try to remedy this problem by adding a latent variable which will conditioned all the decoding process.&lt;/p&gt;

&lt;p&gt;For simplicity, I’ll follow the notation of the paper as much as possible, removing some flourishes to be as clear as possible.&lt;/p&gt;

&lt;p&gt;Let’s start by defining our data:  \((w_1, w_2, …, w_N)\) represents a list of sequences, each sequence contains a series of words \((x_1, x_2, …, x_M)\). N is the number of sequences, and M is the number of words for a given sequence.&lt;/p&gt;

&lt;p&gt;Our goal is to train a model that maximized the probability of our list of sequences. If we have \(S\) list of sequences, we will maximize:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\max \prod_{i=1}^S p({w_1, w_2, .. w_N})&lt;/script&gt;

&lt;p&gt;Here, we write the data distribution as a product of probability as we assume that every list of sequences was generated given the same distribution.&lt;/p&gt;

&lt;p&gt;As we don’t have access to the exact distribution, we will try to infer a parameterized distribution \(p_{\theta}\) represented by our neural network.&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\max \prod_{i=1}^S p_{\theta}({w_1, w_2, .. w_N})&lt;/script&gt;

&lt;p&gt;The logarithm of the probability of multiple joint probabilities simplifies to the sum of the logarithms of the individual probabilities. (Maximizing this sum is also numerically more stable).&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\max log \prod_{i=1}^S p_{\theta}({w_1, w_2, .. w_N}) = \max \sum_{i=1}^S log p_{\theta}({w_1, w_2, .. w_N})&lt;/script&gt;

&lt;p&gt;Let’s try to decompose the term inside the sum:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;log p_{\theta}({w_1, w_2, .. w_N}) = log \sum_{i=1}^Z p_{\theta}({w_1, w_2, .. w_N, z_i})&lt;/script&gt;

&lt;p&gt;&lt;strong&gt;Wait.. what is this z?&lt;/strong&gt; Well in practice we can define a marginal of a random variable as the summation of the joint distribution of this variable and a new random variable. This summation goes over all states of this new variable. In our case, we will refer to z as the latent variable, and \(Z\) the number of state of this variable.&lt;/p&gt;

&lt;p&gt;We have arrived at a more complicated formula that we can’t optimize anymore. If z is continuous, then z has an infinite number of states, and we’ve just made the optimization a little bit hard as we will have to optimize over an infinite sum. Also, we can’t put the log inside the sum for two reasons. From the calculus perspective, it is just incorrect, and from a probabilistic viewpoint, we will have to maximize the joint distribution of the list of sequences and all given states of z. This essentially means that every latent variable \(z_i\) needs to do a good job of explaining the data! Bad.&lt;/p&gt;

&lt;p&gt;One of the most used tricks in calculus is to extend an expression with other terms which keep the first expression correct (like adding 1, subtracting 1).&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;log p_{\theta}({w_1, w_2, .. w_N}) = log \displaystyle{\sum_{i=1}^Z \frac{p_{\theta}({w_1, w_2, .. w_N, z_i}) q_{\beta}(z_i\|w_1, .. w_i)}{q_{\beta}(z_i\|w_1, .., w_i)}}&lt;/script&gt;

&lt;p&gt;Let’s say for now that \(q_{\beta}(z_i|w_1, .. w_i)\) will be a function which will compute the probability of every state of the latent variable at decoding time step \(i\) given the previous utterances and the future one.&lt;/p&gt;

&lt;p&gt;Now because this expression is intractable, we will go for a lower bound of this expression using the Jensens inegality. This relation states that given a concave function \(f\), \(\mathbb{E}_q(f(x)) &amp;gt; f(\mathbb{E}_q(x))\).&lt;/p&gt;

&lt;p&gt;Replacing in our expression, we found that:
&lt;script type=&quot;math/tex&quot;&gt;log \displaystyle{\sum_{i=1}^Z \frac{p_{\theta}({w_1, w_2, .. w_N, z_i}) q_{\beta}(z_i\|w_1, .. w_i)}{q_{\beta}(z_i\|w_1, .., w_i)}} \gt \displaystyle{\sum_{i=1}^Z q_{\beta}(z_i\|w_1, .. w_i) log \frac{p_{\theta}({w_1, w_2, .. w_N, z_i}) }{q_{\beta}(z_i\|w_1, .., w_i)}}&lt;/script&gt;&lt;/p&gt;

&lt;p&gt;We can now decompose our fraction into two terms:
&lt;script type=&quot;math/tex&quot;&gt;\displaystyle{\sum_{i=1}^Z q_{\beta}(z_i\|w_1, .. w_i) log \frac{p_{\theta}({w_1, w_2, .. w_N, z_i}) }{q_{\beta}(z_i\|w_1, .., w_i)}} = 
\displaystyle{\sum_{i=1}^Z q_{\beta}(z_i\|w_1, .. w_i) (log [ p_{\theta}({w_i\|, w_1, .. w_{i-1}, z_i}) + \frac{log p_{\theta}(z_i\|w_1, .. w_{i-1}, )}{q_{\beta}(z_i\|w_1, .., w_i)})}&lt;/script&gt;&lt;/p&gt;

&lt;p&gt;Yeah, we’re getting close! We now have two separate terms. The first term is the conditional likelihood of the next observed utterance given the previous ones, and a sampled z value from the approximate posterior.
The second term is nothing else than the Kullback Leibler divergence between our approximated posterior distribution and the true one. If you want to read more about KL divergence, this &lt;a href=&quot;http://colah.github.io/posts/2015-09-Visual-Information/&quot;&gt;article&lt;/a&gt; is a good place to start. In essence, minimizing this KL divergence will force the approximated distribution to match the real distribution, putting mass at the same points.&lt;/p&gt;

&lt;p&gt;We can adjust our formula to match &lt;sup id=&quot;fnref:vhred:1&quot;&gt;&lt;a href=&quot;#fn:vhred&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;’s expression:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\displaystyle{\sum_{i=1}^Z \mathbb{E}_{z_i \sim q_{\beta}(z_i\|w_1, ..., w_i)} log p_{\theta}(w_i\|w_1, ..., w_i, z_i) - KL(p_{\theta}(z_i\|w_1, .. w_{i-1}) \| q_{\beta}(z_i\|w_1, .., w_i) }&lt;/script&gt;

&lt;p&gt;We have found a lower bound of the log likelihood of our single list of sequence \((w_1, …, w_N)\). 
Because we are using neural networks, the approximated posterior distribution will also be a neural network.&lt;/p&gt;

&lt;p&gt;That’s it for the derivation. 
Now you might ask: How to compute the posterior at generation time?&lt;/p&gt;

&lt;p&gt;Well to my understanding, this is where the math becomes vague. In the paper, during training, they optimized the lower bound of the data likelihood, approximating the posterior of the latent variable given the previous and current utterances. During generation, they conditioned only on the previous utterances. I don’t know why they did so, but it is mathematically incorrect. I guess it was just this moment in deep learning research when “it was just working better.”&lt;/p&gt;

&lt;h2 id=&quot;bibliography&quot;&gt;Bibliography&lt;/h2&gt;

&lt;div class=&quot;footnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:vhred&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1605.06069&quot;&gt; A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues&lt;/a&gt; from &lt;em&gt;Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, Yoshua Bengio&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:vhred&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&amp;nbsp;&lt;a href=&quot;#fnref:vhred:1&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:hred&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;https://arxiv.org/pdf/1507.04808&quot;&gt;Building End-To-End Dialogue Systems Using Generative Hierarchical Neural NetworkModels&lt;/a&gt; from &lt;em&gt;Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville and Joelle Pineau&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:hred&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</description>
        <pubDate>Sat, 29 Jul 2017 21:39:00 -0400</pubDate>
        <link>http://louishenrifranc.github.io/mathblog/2017/07/30/vhred/</link>
        <guid isPermaLink="true">http://louishenrifranc.github.io/mathblog/2017/07/30/vhred/</guid>
        
        <category>math</category>
        
        <category>machine-learning</category>
        
        <category>r</category>
        
        <category>statistic</category>
        
        
        <category>mathblog</category>
        
      </item>
    
      <item>
        <title>On the path towards scaling Bitcoin to billions of transactions. Part I.</title>
        <description>&lt;p&gt;I’ve recently attended a Machine Learning meetup in London, where &lt;a href=&quot;https://iamtrask.github.io/&quot;&gt;i am trask&lt;/a&gt; was presenting his research project, at the intersection of machine learning, homomorphic encryption, and blockchain. His project is to create a platform where a user can share their private data, and let companies train machine learning model on them; and at the end of the journey, the user gets back some money for it, and their data is never shared with anyone. 
If you have the time to read the slides, it is worth it (here is the &lt;a href=&quot;https://docs.google.com/presentation/d/1Tm7jh3pChks0ObJNb7x2ZZhhVOJhmCVN2KDPWyZohQU/&quot;&gt;link&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;At the same time, there have been some amazing features that have just been added to the Bitcoin blockchain such as the lightning network. I decided it was the time to write on BlockChain and crypto currency, as they used to be my area of research two years ago.&lt;/p&gt;

&lt;p&gt;I’m going to present briefly what a Blockchain is, what Bitcoin is, and how both of them works? If you feel the need to deepen your understanding of Bitcoin/blockchain, I still think this &lt;a href=&quot;Mastering Bitcoin&quot;&gt;book&lt;/a&gt; from Andreas M. Antonopoulos is a reference. I loved the book, the writer is amazing, and I love his state of mind.&lt;/p&gt;

&lt;p&gt;So let’s go. :)&lt;/p&gt;

&lt;p&gt;Just for the sake of simplicity, when I’ll refer to Blockchain in the future, I’ll be talking about the Bitcoin’s Blockchain!&lt;/p&gt;

&lt;h1 id=&quot;what-is-a-blockchain&quot;&gt;What is a Blockchain?&lt;/h1&gt;
&lt;p&gt;&lt;img src=&quot;https://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Blockchain.svg/102px-Blockchain.svg.png&quot; alt=&quot;Figure 1. A representation of a blockchain with multiple forks&quot; /&gt;&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;It is an underlying protocol of the current implementation of the famous crypto currency Bitcoin. There is no Bitcoin without Blockchain. There could be Blockchain without Bitcoin.&lt;/li&gt;
  &lt;li&gt;A blockchain is a distributed data base, which handles a list of records protected against falsification and modification by the nodes that stored it. Multiple entities can hold the Blockchain, but no entry can be faked. Of course, anyone can try to tamper their local version of the Blockchain, but it will instantly become desynchronized with the rest of the network.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;what-is-a-block&quot;&gt;What is a block?&lt;/h1&gt;
&lt;p&gt;&lt;img src=&quot;http://orm-chimera-prod.s3.amazonaws.com/1234000001802/images/msbt_0701.png&quot; alt=&quot;Figure 2: An example of a block &quot; /&gt;
Block - Chain. In Blockchain, you add blocks. But what are these blocks? Well for the Bitcoin Blockchain, a block is composed of:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;A list of transaction that the block verify. When the block is added to the blockchain, this transaction will be considered made, just as when you pay in the supermarket with your credit card.&lt;/li&gt;
  &lt;li&gt;A header identifying the block uniquely. This header contains:
    &lt;ul&gt;
      &lt;li&gt;The version of the Blockchain&lt;/li&gt;
      &lt;li&gt;A unique block hash. The block hash identifies a block uniquely and unambiguously and can be independently derived from any node by simply hashing the block header.&lt;/li&gt;
      &lt;li&gt;A unique hash identifying the previous block. Same principle. Because blocks are stacked on top of each other, when miners create a new block, they mine on top of another extracted block.&lt;/li&gt;
      &lt;li&gt;A Merkle Tree&lt;/li&gt;
      &lt;li&gt;A timestamp&lt;/li&gt;
      &lt;li&gt;A nonce&lt;/li&gt;
      &lt;li&gt;A level of difficulty.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Merkle Tree, Nonce, Level of difficulty, let me explain them a little bit more :)&lt;/p&gt;

&lt;h2 id=&quot;what-is-a-merkle-tree&quot;&gt;What is a Merkle Tree?&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;http://orm-chimera-prod.s3.amazonaws.com/1234000001802/images/msbt_0703.png&quot; alt=&quot;Figure 3. A simple Merkle Tree with four transactions&quot; /&gt;
A Merkle tree, also known as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Tree_(data_structure)&quot;&gt;binary hash tree&lt;/a&gt;, is a data structure used for efficiently summarizing and verifying the integrity of large sets of data. In the blockchain, transactions are saved in the Merkle Tree. The tree is constructed in a bottom-up manner when leaves contain a unique representation of a single Bitcoin transaction, a hash. The top node is used to compute the single block hash.&lt;/p&gt;

&lt;p&gt;The Merkle Tree builds on top of this leaves, and the root of the tree becomes a unique identifier of all these transactions. 
The use of the data structure, is to allow certain nodes that do not hold a version of the blockchain to be able to verify the correctness of any past
transaction in a fraction of exchanges. For example, given a transaction, and a unique block hash, anyone can re-compute the path locally from the leave to the top nodes by asking for adjacent nodes (blue node in the next figure).&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;http://orm-chimera-prod.s3.amazonaws.com/1234000001802/images/msbt_0705.png&quot; alt=&quot;How Merkle Tree can be used for efficiently verifying any transaction&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Proof of Concept for building a Merkle Tree in C++&lt;/strong&gt;&lt;/p&gt;
&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;void Block::BuildMerkleRoot()
{
    // To get a even number of transactions
    if (transactions.size() &amp;amp; 1) {
        transactions.push_back(transactions.at(transactions.size() - 1)); nombreTransaction++;
    }
    int N = transactions.size();

    vector&amp;lt;string&amp;gt; hashTree;
    hashTree.resize(2 * N - 1);
    for (int i = 0; i &amp;lt; N; i++)
        hashTree.at(2 * N - 2 - i) = transactions.at(i);
    for (int i = N - 2; i &amp;gt; -1; i--)
    {
        // Double SHA256 of the nodes
        hashTree.at(i) = SHA25::sha256(SHA25::sha256(hashTree.at(2 * i + 1) + hashTree.at(2 * i + 2)));
    }
    
    // Set the Hash Merkle Root in the header
    header.setHashMerkleRoot(hashTree.at(0));
    // Set the block Hash
    header.setBlockHash(SHA25::sha256(SHA25::sha256(header.get_HashMerkleRoot())));
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h1 id=&quot;what-mining-the-blockchain-is&quot;&gt;What mining the Blockchain is?&lt;/h1&gt;
&lt;p&gt;First of all, to make things clear, only a part of the bitcoin network is mining! They do the hard job of keeping the blockchain secure, while as bitcoin user we can safely make the transaction between peers. Without mining, there is no safe blockchain.&lt;/p&gt;

&lt;p&gt;Think about mining as a way to assess that some work has been made (i.e., a proof of work). Because mining is computationally expensive, if you can solve the mining problem, you’ll get a reward for it, here bitcoin. That’s is one way for the miner to get paid (not the only one).&lt;/p&gt;

&lt;p&gt;Here is the mining problem:
Find a nonce &lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;\(n\)&lt;/span&gt; for a given block &lt;span style=&quot;color:green; font-weight: bold;&quot;&gt;\(B\)&lt;/span&gt;, and a difficulty &lt;span style=&quot;color:yellow; font-weight: bold;&quot;&gt;\(D\)&lt;/span&gt;, such that:&lt;/p&gt;

&lt;p&gt;SHA256(SHA256(headerHash(&lt;span style=&quot;color:green; font-weight: bold;&quot;&gt;\(B\)&lt;/span&gt; + &lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;\(n\)&lt;/span&gt;))) &amp;lt; 0.(&lt;span style=&quot;color:yellow; font-weight: bold;&quot;&gt;\(D\)&lt;/span&gt; zeros)1000000&lt;/p&gt;

&lt;p&gt;SHA256 is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Hash_function&quot;&gt;hash function&lt;/a&gt;, it is a noninvertible function. It means that if \(f(x) = y\), and \(y\) is known, there is no finite form to compute back what was \(x\). Then, if you want to find out what this \(x\) is, you’ll better start off now because every possible value in the input space can be the correct one. The solution of this problem is completely random.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Proof of Concept for mining in C++&lt;/strong&gt;&lt;/p&gt;
&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;paire Block::solveProofofWork()
{
    unsigned long long nonce = 0, incr = 0;
    unsigned long long const limit = std::numeric_limits&amp;lt;unsigned long long&amp;gt;::max();

    // Here we set the mining difficulty to finding a hash where the first Constant::DIFFICULTY_MINING are zeros.
    string sol(Constant::DIFFICULTY_MINING, '0');
    // Until no solution has been found
    while (true) {
        // Compute a new hash
        string hash = SHA25::sha256(string(header + std::to_string(incr) + std::to_string(nonce)));
        if (hash.substr(0, Constante::DIFFICULTY_MINING) == sol)
            break;
        else
            ++nonce;
        if (limit - 1 == nonce) // Increment the incr when nonce reach unsigned long long int max value
        {    
            // To overcome machine precision limit, if we reach the larget int, we add an increment, which would 
            // modify the hash
            incr++;
            nonce = 0;
        }
    }
    header.setNonce(paire(incr, nonce));

    return paire(incr, nonce);
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h4 id=&quot;why-are-we-calling-it-a-proof-of-work&quot;&gt;Why are we calling it a proof of work?&lt;/h4&gt;
&lt;ul&gt;
  &lt;li&gt;The number of trailing zeros is currently 14, which corresponds to 10 minutes of computation for a supercomputer, but an eternity for any modern laptop.&lt;/li&gt;
  &lt;li&gt;Every 2000 blocks, the mining difficulty is automatically recomputed.&lt;/li&gt;
  &lt;li&gt;There are approximately 1.000.000.000.000.000.000.000.000 hashes to compute per block.&lt;/li&gt;
  &lt;li&gt;A miner with a hash rate of 1 trillion hashes per second would take 59 days to solve the proof of work&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;why-this-computation-problem-make-the-blockchain-secure&quot;&gt;Why this computation problem make the Blockchain secure?&lt;/h4&gt;
&lt;p&gt;Forging fake blocks is impossible! Remember, &lt;em&gt;a block is correct if it has a reference to a previously mined block&lt;/em&gt;. When you search for the correct nonce, you are hashing a mixture of your new block’s header, containing a reference to the previous block hash, and this nonce. If you succeed to find the correct nonce, then you can share it with the community and anyone can checks if the new block is valid. Hence it is unlikely that any miner will find the nonce multiple times in a row before everyone.&lt;/p&gt;

&lt;h1 id=&quot;what-are-bitcoin-forks&quot;&gt;What are Bitcoin forks?&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;A fork appears when two miners solve the proof of work at the same time in different part of the network.&lt;/li&gt;
  &lt;li&gt;They propagate their new block to their neighbors. For simplicity, imagine that half of the network receives the first on  (&lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;red&lt;/span&gt; block), the other half, the southern hemisphere receives the other one, the &lt;span style=&quot;color:green; font-weight: bold;&quot;&gt;green&lt;/span&gt; block.&lt;/li&gt;
  &lt;li&gt;Every node verify the incoming block. If the block is valid, they add it to the blockchain and pass it to their neighbors.&lt;/li&gt;
  &lt;li&gt;The network is now probably cut in two half, where the last block added differs.&lt;/li&gt;
  &lt;li&gt;Half of the miner will decide to mine on top of the &lt;span style=&quot;color:green; font-weight: bold;&quot;&gt;green&lt;/span&gt; block, the other part, on the &lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;red&lt;/span&gt; block.&lt;/li&gt;
  &lt;li&gt;If a new block (&lt;span style=&quot;color:yellow; font-weight: bold;&quot;&gt;yellow&lt;/span&gt;) block is actually mined  first on top of the &lt;span style=&quot;color:green; font-weight: bold;&quot;&gt;green&lt;/span&gt; one, it will spread again across the network. During that time, both &lt;span style=&quot;color:green; font-weight: bold;&quot;&gt;green&lt;/span&gt; and &lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;red&lt;/span&gt; will have been shared to all peers of the network, so “&lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;red&lt;/span&gt; miner” will realise they received a new block that has a higher height. At this moment they can stop mining on top of the &lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;red&lt;/span&gt; block, and restart again with the &lt;span style=&quot;color:yellow; font-weight: bold;&quot;&gt;yellow&lt;/span&gt; one as predecessor, or keep trying to solve the POF with the &lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;red&lt;/span&gt; block as the previous one. If they do so, it might appear that this path will be privileged, because some miners were able to mine block quicker on top of this fork.&lt;/li&gt;
  &lt;li&gt;Every mining pool has its optimal way of splitting the mining work, but often, they tried to build different block on top of different branch, to potentially maximize their chance of being the first to mine a block that will get accepted as in the main Blockchain.&lt;/li&gt;
  &lt;li&gt;Given that the miners are always mining on top of the longest chain, a common consensus is established beyond a certain depth.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;on-the-recent-improvement-of-the-bitcoin&quot;&gt;On the recent improvement of the Bitcoin&lt;/h1&gt;
&lt;p&gt;This common consensus can take up to one hour before they are irreversible. A new network build on top of the bitcoin has recently been merged to the core code. The Lightning Network is a decentralized system for instant, high-volume micropayments that removes the risk of delegating custody of funds to trusted third parties. In a next article, I’ll explain in depth what are the benefit of this new implementation.&lt;/p&gt;
</description>
        <pubDate>Fri, 14 Jul 2017 21:39:00 -0400</pubDate>
        <link>http://louishenrifranc.github.io/techblog/2017/07/15/blockchain/</link>
        <guid isPermaLink="true">http://louishenrifranc.github.io/techblog/2017/07/15/blockchain/</guid>
        
        <category>math</category>
        
        <category>machine-learning</category>
        
        <category>r</category>
        
        <category>statistic</category>
        
        
        <category>techblog</category>
        
      </item>
    
      <item>
        <title>From the origin of Generative Modeling to the recent Generative Adversarial Network.</title>
        <description>&lt;p&gt;Here is a presentation on Generative Adversarial Network.&lt;br /&gt;
The goal of this presentation was to review the GAN framework in depth, by first looking at the original formulation by &lt;em&gt;Ian Goodfellow &amp;amp; al&lt;/em&gt;, and then, present some recent results.&lt;/p&gt;

&lt;style&gt;
.responsive-wrap iframe{ max-width: 100%;}
&lt;/style&gt;

&lt;div class=&quot;responsive-wrap&quot;&gt;
&lt;!-- this is the embed code provided by Google --&gt;
&lt;iframe src=&quot;https://docs.google.com/presentation/d/1lE-rz5MAhSthlbJX4fPTwgo0XvGdsHILp-OrVWLqAck/embed?start=false&amp;amp;loop=true&amp;amp;delayms=30000&quot; frameborder=&quot;0&quot; width=&quot;1440&quot; height=&quot;839&quot; allowfullscreen=&quot;true&quot; mozallowfullscreen=&quot;true&quot; webkitallowfullscreen=&quot;true&quot;&gt;&lt;/iframe&gt;
&lt;!-- Google embed ends --&gt;
&lt;/div&gt;
</description>
        <pubDate>Fri, 30 Jun 2017 21:39:00 -0400</pubDate>
        <link>http://louishenrifranc.github.io/mathblog/2017/07/01/gan_overview/</link>
        <guid isPermaLink="true">http://louishenrifranc.github.io/mathblog/2017/07/01/gan_overview/</guid>
        
        <category>math</category>
        
        <category>machine-learning</category>
        
        <category>gan</category>
        
        
        <category>mathblog</category>
        
      </item>
    
      <item>
        <title>On recent progress in dialogue generation</title>
        <description>&lt;p&gt;At &lt;a href=&quot;http://trueai.io/&quot;&gt;True Ai&lt;/a&gt;, we apply state of the art machine learning models to improve our product helping agents in customer support to answer questions faster and better. As I recently started as an intern, I’ve spent some of my free time reading the literature. Here is a nonexhaustive summary of NLP papers I found relevant.&lt;/p&gt;

&lt;p&gt;This article is quite a mess. I tried to keep as much structure for every review. Hopefully you’ll find something useful out of this jungle.&lt;/p&gt;

&lt;h1 id=&quot;how-not-to-evaluate-your-dialogue-system-an-empirical-study-of-unsupervised-evaluation-metrics-for-dialogue-response-generation&quot;&gt;How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Papers&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1603.08023.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2016&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems&quot;&gt;Problems&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;Evaluating the quality of generated answers&lt;/strong&gt; from a seq2seq model is an open problem. Comparison of different metrics and different models used in the context of dialogue systems.&lt;/p&gt;

&lt;h3 id=&quot;solutions&quot;&gt;Solutions&lt;/h3&gt;
&lt;p&gt;No real solution to this problem, only a review of the metrics used.&lt;br /&gt;
Here is the word based metrics compared:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;BLEU&lt;/li&gt;
  &lt;li&gt;METEOR&lt;/li&gt;
  &lt;li&gt;ROUGE&lt;br /&gt;
Here are the embedding based metrics compared:&lt;/li&gt;
  &lt;li&gt;Mean of word embedding,&lt;/li&gt;
  &lt;li&gt;Mean combined with TF-IDF (This one was not presented in the paper, but it is currently state of the art, see &lt;a href=&quot;https://openreview.net/pdf?id=SyK00v5xx&quot;&gt;here&lt;/a&gt;)&lt;/li&gt;
  &lt;li&gt;Greedy matching. It computes the average sum of of cosine similarity between pair of words in both sentences, pairs matched greedily&lt;/li&gt;
  &lt;li&gt;Vector extrema. In each dimension of word embeddings, take the largest value to get the sentence embedding&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here are the retrieval based model compared:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;TF-IDF: In the context of retrieval based model, every context, and response are encoded, and then responses are compared and ranked using cosine similarity.&lt;/li&gt;
  &lt;li&gt;Dual encoder: Have two RNN encode questions and answers respectively, both outputting a final embedding. Then, compute a probability that a reply \(a\) is good for a question \(q\) using \(p(a|q) = \sigma(q^T M a + bias)\). Trained with negative sampling (“word2vec loss”).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here are the generative model compared:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;LSTM: simple LSTM model&lt;/li&gt;
  &lt;li&gt;HRED: See my review on it&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;results&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: Twitter corpus and Ubuntu Dialogue Corpus&lt;/li&gt;
  &lt;li&gt;BLEU-3 has the best correlation with the Human judgment on dialogue answer generation. Better than METEOR, ROUGE, and all embeding based metrics.&lt;/li&gt;
  &lt;li&gt;Best performance of HRED when used with mean of word embedding.&lt;/li&gt;
  &lt;li&gt;HRED Better than LSTM everytime on Twitter and Ubuntu corpus.&lt;/li&gt;
  &lt;li&gt;TF-IDF not good performance, Dual encoder is better.&lt;/li&gt;
  &lt;li&gt;Embedding metrics: mean of word embedding is good, vector extrema are bad.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;mutual-information-and-diverse-decoding-improve-neural-machine-translation&quot;&gt;Mutual Information and Diverse Decoding Improve Neural Machine Translation&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Papers&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1601.00372.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2016&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-1&quot;&gt;Problems&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;Vanilla seq2seq ignores the probability \(p(question|answer)\). In the context of translation, it has been proved that this information was relevant for the pertinence of the answer (See &lt;a href=&quot;http://www.mitpressjournals.org/doi/pdf/10.1162/coli_a_00015&quot;&gt;here&lt;/a&gt;). In &lt;a href=&quot;https://arxiv.org/abs/1510.03055&quot;&gt;there&lt;/a&gt;, they came with a formula computing the best answer using \(p(q|a)\) using Mutual Maximum Information and deriving their formula with Bayes theorem. However, as mentioned in the paper, it is impossible to compute this distribution in a sequential model like seq2seq, because you don’t know the answer until you’ve decoded it.&lt;/li&gt;
  &lt;li&gt;Beam search usually returns very similar answer.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;solution&quot;&gt;Solution&lt;/h3&gt;
&lt;h4 id=&quot;problem-1&quot;&gt;Problem 1&lt;/h4&gt;
&lt;p&gt;They trained two seq2seq models asynchronously:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;The first one \(S2S_f\) returns a list of potential answer, each one has been selected by a variant of beam search (see &lt;strong&gt;Problem 2)&lt;/strong&gt;).  It allows them to compute the log likelihood of an answer given the question.&lt;/li&gt;
  &lt;li&gt;The second seq2seq model \(S2S_b\) is trained to predict the question that would have to generate an answer. It can be seen as a reversed seq2seq model. This model allow use to compute \(log p(x|y)\) for all \(y\) in the beam-search list. This score is added to the final cost function (multiplied by a regularizer term) and allows to rank predicted answers.&lt;/li&gt;
  &lt;li&gt;Finally they also added a regularizer term for the length \(L_t\) of the sequence \(\nu L_t\).&lt;br /&gt;
The final cost consists of maximizing all this three term summed.&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;problem-2&quot;&gt;Problem 2:&lt;/h4&gt;
&lt;p&gt;For this second issue, the classic beam search optimization problem formulation has been a little bit transformed. During decoding, at each time-step \(t\), a penalization term (\(\lambda rank_t\)) is added to the logprob, where \(rank_t\) is the rank of the t-th most probable token in the beam search. So if beam-search size is \(K\), a re-ranking is done on the top-prediction of the decoder for every lists \([y_1^1, … y_{t-1}^1], [y_1^2, … y_{t-1}^2], …, [y_1^K, … y_{t-1}^K]\). Then, every prediction added to its score the opposite of its rank \(- \lambda rank_t\). Because the goal is to maximize, low ranked prediction are not privileged.&lt;/p&gt;
&lt;h3 id=&quot;results-1&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: English German dataset&lt;/li&gt;
  &lt;li&gt;Got better BLEU score on training (~+4 in BLEU score between standart seq2seq and model trained with the new loss function)&lt;/li&gt;
  &lt;li&gt;For translation tasks, the average number of distinct unigrams and bigrams in the returned list of sequences doubled (from 0.5 to 1, and 1.5 to 2.8 for bigrams)&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;sequence-to-backward-and-forward-sequences-a-content-introducing-approach-to-generative-short-text-conversation&quot;&gt;Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1607.00970.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2016&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-2&quot;&gt;Problems&lt;/h3&gt;
&lt;p&gt;Answer generated by seq2seq model are too universally relevant, usually also too “sage” (Yes, No….)&lt;/p&gt;

&lt;h3 id=&quot;solutions-1&quot;&gt;Solutions&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;For a given question, defined as a list of tokens \(q = (q_1, q_2, … q_{|Q|}) \), they computed the &lt;em&gt;PointWise Mutual Information&lt;/em&gt; of all words in the question with every other words \(w_r\) in the decoder vocabulary. From this comparaison, they selected the word \(w_r\) equals to \(argmax( \sum_{i=1}^{|Q|}log \frac{p(w_i|w_r)}{p(w_i)})\). This word can be seen as the most important word in the answer.&lt;/li&gt;
  &lt;li&gt;Then they used two seq2seq model; let’s called them \(S_b\), and \(S_f\). With the word \(w_r\) and \(S_b\)’s decoder, they generated in reverse order what will come in the sentence before \(w_r\). With \(S_f\)’s decoder, and the sequence generated, they predicted what will be after \(w_r\). To summarize, \(S_f\) predicts words before \(w_r\), and \(S_b\) those after. Both seq2seq models see and encode the question.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;results-2&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: Human conversation from Baidu Tieba forum&lt;/li&gt;
  &lt;li&gt;Vanilla seq2seq generates longer sentences (5.61 versus 5.3 words)&lt;/li&gt;
  &lt;li&gt;Human evaluation gives better score to sequence generated using their method (0.67 versus 0.58)&lt;/li&gt;
  &lt;li&gt;Entropy for the answer is greater for their model than vanilla seq2seq.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;a-simple-fast-diverse-decoding-algorithm-for-neural-generation&quot;&gt;A Simple, Fast Diverse Decoding Algorithm for Neural Generation&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1611.08562.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2016&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-3&quot;&gt;Problems&lt;/h3&gt;
&lt;p&gt;Not a lot of diversity in neural generation. Beam search gives a small list of redundant predictions compared to all the possibilities that could have been explored.&lt;/p&gt;

&lt;h3 id=&quot;solutions-2&quot;&gt;Solutions&lt;/h3&gt;
&lt;p&gt;See solution of Problem 2 for Mutual Information, and Diverse Decoding Improve Neural Machine Translation. It is the same global idea, but in this paper, they made \(\lambda\) a non-hyperparameter, by using a Policy Gradient algorithm to train a policy to predict the good \(lambda\) for input.&lt;/p&gt;

&lt;h3 id=&quot;results-3&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: OpenSubtiles&lt;/li&gt;
  &lt;li&gt;They used BLEU score. Having a reinforcement learning predict lambda seems to &lt;strong&gt;slightly&lt;/strong&gt; better results than with fixing lambda. The greatest difference between those two techniques in all the experiments is 6% more in BLEU score (baseline 1.88, no RL: 2.21, with RL: 2.32).&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;neural-conversational-model-with-mutual-information-ranking&quot;&gt;Neural Conversational Model with Mutual Information Ranking&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://web.stanford.edu/class/cs224n/reports/2761918.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2016&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-4&quot;&gt;Problems&lt;/h3&gt;
&lt;p&gt;The ”I don’t know” problem and de-prioritizing over-generic responses&lt;/p&gt;

&lt;h3 id=&quot;solutions-3&quot;&gt;Solutions&lt;/h3&gt;
&lt;p&gt;They trained a seq2seq model with soft-attention and a loss function for re-ranking beam search results. It is based on Mutual Information formula. After deriving their maths, they arrived to: \(r_i = arg max P(r|s_i)(ln(P(r|s_i) - ln(P(r)))\), which is different from the &lt;em&gt;A Diversity-Promoting Objective Function for Neural Conversation Model&lt;/em&gt;. The first two terms are computing while decoding. Instead, the last one is precomputed, using a larger dataset.&lt;/p&gt;

&lt;h3 id=&quot;results-4&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: Ubuntu&lt;/li&gt;
  &lt;li&gt;Well not a good paper, badly written. Results and experiments are not the ones we usually expected. They gave some samples, and it &lt;strong&gt;seems&lt;/strong&gt; to produce more different answer.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;a-diversity-promoting-objective-function-for-neural-conversation-model&quot;&gt;A Diversity-Promoting Objective Function for Neural Conversation Model&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1606.03352.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2016&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-5&quot;&gt;Problems&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;the ”I don’t know” problem and de-prioritizing over-generic responses&lt;/li&gt;
  &lt;li&gt;Introducing another metric than perplexity for measuring diversity in generated sentences&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;solutions-4&quot;&gt;Solutions&lt;/h3&gt;
&lt;h4 id=&quot;problem-1-1&quot;&gt;Problem 1&lt;/h4&gt;
&lt;p&gt;Started from the Maximum Mutual Information formula, they derived two models. In an incomplete approximated Bayesian framework, they took into account the marginal probability of an answer, as a regularizer term against generic responses. Here are the two models:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;MMI-antiLM: First model consists of a new score function to choose during beam search: \(log p(A|Q) - \lambda U(A)\) where \(U(A) = 1\text{ if }|A| &amp;lt; k\), k choose equal to 5.&lt;/li&gt;
  &lt;li&gt;MMI-bidi: Second model does beam search until the end, then reranks answers using a scoring function:  \((1 - \lambda) log p(A|Q) + \lambda p(Q|A)\). The first term is the standard loss function, the second one, well I don’t know how they computed it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&quot;problem-2-1&quot;&gt;Problem 2&lt;/h4&gt;
&lt;p&gt;They also introduced a new metric for computing diversity: &lt;em&gt;diversity-1&lt;/em&gt;, and &lt;em&gt;diversity-2&lt;/em&gt;, respectively the number of unigrams, bi-grams in the generated sentences, divided by the number of tokens in the sentences.&lt;/p&gt;

&lt;h1 id=&quot;a-practical-approach-to-dialogue-response-generation-in-closed-domains&quot;&gt;A practical approach to dialogue response generation in closed domains&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Papers&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1703.09439.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2017&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-6&quot;&gt;Problems&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;Each agent working in Amazon has a list of blurbs accessible to copy and paste to answer client questions. However, there are a lot of theses, and different agents may use more often different ones. Hence it would be good to have a list of templated answer based on the history of the agent.&lt;/li&gt;
  &lt;li&gt;Now that we have a proper list of templated answer for every agent, the goal is to find the best one given a client query.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;solutions-5&quot;&gt;Solutions&lt;/h3&gt;
&lt;h4 id=&quot;problem-1-2&quot;&gt;Problem 1&lt;/h4&gt;
&lt;p&gt;It isn’t answered in the paper, but question is at least raised.&lt;/p&gt;
&lt;h4 id=&quot;problem-2-2&quot;&gt;Problem 2&lt;/h4&gt;
&lt;ol&gt;
  &lt;li&gt;Train a dual encoder network to encode a question \(q\), and an answer \(a\). Then try to predict the best response by ranking them using a score function: \(\sigma(q^T M a + bias)\). The model is trained to predict the best answer using negative sampling (the ratio of positive-negative pairs for training is 1:2).&lt;/li&gt;
  &lt;li&gt;Sample 400K answers from a dataset, compute a reply embedding by retrieving the output of the trained dual encoder. Cluster embeddings into 500 ones using mini batch K-mean with K-means++ initialization. Each cluster is assigned a template answer that is the one that has the closest embedding from the centroid. With this initial 500 groups, human evaluations pick 200 of them.&lt;/li&gt;
  &lt;li&gt;During inference, every 200 templated answer is compared with the question using the dual encoder. The answers giving the top-k highest probs are returned.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;results-5&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: Amazon internal item delivery issue&lt;/li&gt;
  &lt;li&gt;Response suggestions are more relevant than TF-IDF methods (Human evaluation).&lt;/li&gt;
  &lt;li&gt;100 questions: In top 3 answers returned, model found a very relevant one for 48 questions, TF-IDF only 31.&lt;/li&gt;
  &lt;li&gt;Precision@3: 80% for their model, 65% for the tf-idf.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;a-deep-reinforced-model-for-abstractive-summarization&quot;&gt;A Deep Reinforced Model for Abstractive Summarization&lt;/h2&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Papers&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1705.04304.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2017&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-7&quot;&gt;Problems&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;Attentional RNN-based encoder-decoder models for abstractive summarization have achieved good performance on short input and output sequences. However, for longer documents, these models perform badly and often include repetitive phrase.
Also, models are trained only with the former often exhibit “exposure bias”, because ground truth is often provided while training.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;solutions-6&quot;&gt;Solutions&lt;/h3&gt;
&lt;p&gt;Model is based on the standard seq2seq encoder decoder probabilistic model. The “vanilla way” of doing attention is to compute att vectors at each decoding step t (vector \(c_t^e\)). However, this form of attention does not prevent the model for generating repetitive phrases in the generated summary.  &lt;br /&gt;
A proposed solution is to do an &lt;strong&gt;intra decoding attention&lt;/strong&gt;, introducing a new attention vector \(c_t\) for each decoding step. This attention mechanism looks at previous hidden state \(h_{t^1}\) of the decoding sequence. Here are the formula:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;Attention vectors: \(e_{tt^\prime} = h_t W_{att}^{dec} h_{t^\prime}\),&lt;/li&gt;
  &lt;li&gt;Weighted coefficient: \(\alpha_{tt^\prime} = \frac{exp(e_{tt^\prime})}{\sum_{j} \exp(e_{tt^j})}\),&lt;/li&gt;
  &lt;li&gt;Intra decoding vector: \(c_t^d = \sum_j (\alpha_{t^j} e_{tt^j})\). This context vector is used when generating a new token.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;While generating a new token, they allowed tokens from the input sequence to be copied by using a differentiable copying mechanism: at each decoding time-step \(t\), the model could decide to copy a token from the input or do a classical softmax.     &lt;br /&gt;
The decision variable for copying from input is \(u_t\). Here is how it is computes: \(u_t = 1\) with probability \(p(u_t = 1) = \sigma(W [h_t||c_t^e||c_t^d] + b)\).&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;If \(u_t = 1\), then they choosed a token from the input, using the already computed weighted coefficient \(\alpha_t^e\) over the input sequence attention coefficient.&lt;/li&gt;
  &lt;li&gt;Otherwise, if \(u_t = 0\), they used the classical softmax formula which take as input the hidden state \(h_t\), the input sequence attention vector \(c_t^e\), and the intra decoding vectors \(c_t^d\).&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;efficient-natural-language-response-suggestion-for-smart-reply&quot;&gt;Efficient Natural Language Response Suggestion for Smart Reply&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1705.00652.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-8&quot;&gt;Problems&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;End-to-end models are difficult to train: model needs to learn the language, but also to do “something with it.”&lt;/li&gt;
  &lt;li&gt;End-to-end models, like seq2seq, needs to be consistent in a dialogue with a human, maintaining some memory&lt;/li&gt;
  &lt;li&gt;While seq2seq model provided a generalized solution to answer generation, training is long and hard&lt;/li&gt;
  &lt;li&gt;seq2seq model are generative model, and after generating answer, we need a biased method to rank answers&lt;br /&gt;
 This paper is base on the original &lt;em&gt;SmartReply&lt;/em&gt; paper. In SmartReply, an initial neural network is trained to predict whether an email will be answered by a short answer. If the nn triggered ‘yes,’ then a list of possible answers is retrieved from a database using a seq2seq, with beam search, trained to maximize the conditional distribution of the reply given the question. To ensure diversity in the predicted answer, a clustering algorithm is applied to all answer to ensure no duplicate answers are provided. Furthermore, because all answers are labeled, before returning the top 3 answers, they ensure at least a negative and positive suggestions.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;solutions-an-overview-of-the-model&quot;&gt;Solutions… an overview of the model&lt;/h3&gt;
&lt;p&gt;Instead of using a seq2seq model, they went for a feedforward neural network. The model try to find the \(argmax P(y|x) = \frac{P(x, y)}{\sum_{i=1}^K P(x, y)}\), where \(K\) is a number of response sampled to compute the conditionnal probability. Sampling is done uniformly and is used to reduce the computation bottleneck. The joint distribution \(P(x, y) = exp(S(x,y))\)is computed by a feedforward network. For faster inference, the model doesn’t modelize the all joint distribution, instead question and answer are passed as different input of a neural network, and information flow is combined only at the last layer, as a dot product between previous output. It returns a score function of how likely \(x, y\) should be paired, defined as \(S(h_x,h_y)\).&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;The embeddings&lt;/strong&gt;: Answers and questions are computed using a combination of feature hashing, n-gram word representation, and word representation sums.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Loss function&lt;/strong&gt;: In a batch, every \((x, y)\) corresponds to a real pair fo question-answer. For a \(x_i\), all other \(y_j, j \neq i\) are used as negative candidates. Minimizing this loss function is the training goal:
      \(L(x, y) = -\frac{1}{K} \sum_{i=1}^K log P(y_i|x_i) = -\frac{1}{K} \sum_{i=1}^K  S(x, y) - log \sum_{j=1, j \neq i}^{K-1} exp(S(x_i, y_j))\)&lt;/li&gt;
  &lt;li&gt;Dealing with email systems, it is usually interesting to encode an email (i.e \(x\)), as a combination of multiple features. Hence for a single example, multiple features (i.e \(x_i\text{ ,} i \in |F|\)) are extracted from it. For these reason, \(i\) different models, with the same architecture, are created, each one returning a single score \(S(x_i, y)\), a response output vector \(h_{y^i}\text{ ,}i \in |F|\), and a feature output vector \(h_{x^i}\). Afteward, the mean of \(h_{x^i}\), and \(h_{y^i}\) is computed, and another new neural network with the same architecture, but where input are the mean of the responses output and the features output vector.  The output is a single \(h_x\) and also the response vector \(h_y\).&lt;br /&gt;
Both the first neural network and the last one are trained jointly, and the final loss resulting is: \(= L(x, y) + \sum_{i=1}^{|F|} L(x_i, y)\).&lt;/li&gt;
  &lt;li&gt;During training, the approximation of the denominator leads to a bias estimation of the denominator, because common responses appearing more in the training data are often selected as negative sampled, as a result during inference, the model often returns long answers. One solution is to include a penality term \(+\alpha P_{LM}(y)\) to the final score. This penalization term is towards providing more generic answers&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;results-6&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: Google internal dataset, 300 million pairs of questions answers.&lt;/li&gt;
  &lt;li&gt;The joint neural networks outperform the other one (the one explained) on @precision1 given 100  answers, one right. But if we increase the batch size during training, the joint neural network is longer to train, and then it is better to use the one explained (independent model).&lt;/li&gt;
  &lt;li&gt;Model is not way better than the classical seq2seq model. However, it is way faster (100%)&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;end-to-end-memory-networks&quot;&gt;End to end memory networks&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;http://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-9&quot;&gt;Problems&lt;/h3&gt;
&lt;p&gt;The first paper of Memory Network was hard to train and required supervised help.&lt;/p&gt;

&lt;h3 id=&quot;solutions-7&quot;&gt;Solutions&lt;/h3&gt;
&lt;p&gt;The problem consists in predicting an answer \(a\) given a context (list of sentences \((x_0, …, x_i)\)), and a query \(q\).&lt;br /&gt;
The model is very similar to the original memory network architecture.&lt;/p&gt;

&lt;p&gt;First, every sentence embeddings are computed as a weighted sum of their word embedding. For the sentence embedding, they went for a crazy weighting, that I’m personally not convinced about, but here is the formula: \(l_{jk} = ((1 - \frac{j}{J}) - \frac{k}{K}(1 - \frac{2j}{J})\), where \(j\) is the position of the word in the sentence, and \(k\) is the number of dimension of the word embeddings.&lt;/p&gt;

&lt;p&gt;There are two main embedding matrices:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;\(A\) transforms input into embeddings. Those embeddings are used to compute the match between a query and the input&lt;/li&gt;
  &lt;li&gt;\(C\) is only used to update our query vector. Theses embeddings are only used to update the query, not to try to find matches between inputs and the query.&lt;br /&gt;
Now, here are the formula:&lt;/li&gt;
  &lt;li&gt;\(s_i = \sum_{j}^{nbwords} l_j A[x_j]\), and \(c_i = \sum_{j}^{nbwords} l_j C[x_j]\)&lt;/li&gt;
  &lt;li&gt;The query \(q\) is also transformed in a vector \(u^0\) in the same way as other sentences.&lt;/li&gt;
  &lt;li&gt;Match between the query vector \(u^0\) and each sentence embedding \(s_i\) is computed as a dot product, followed by a softmax to have a weighted probability \(p_i\), corresponding to the importance of each input.&lt;/li&gt;
  &lt;li&gt;This weighting is used to create a new output vector \(o_{t} = sum_i^{nbwords} p_i c_i\), which will be linearly transformed with previous time-step \(u_{t-1}\) (i.e \(u_{t} = W(o_t + u_{t-1})\)).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;\(t\) represents the number of times this “memory lookup” is done, it is often referred as the number of hops.&lt;/p&gt;

&lt;p&gt;The difference between this paper and the previous M2M paper is that they used a softmax instead of the real max to retrieve some information from the sentences inputs.&lt;/p&gt;

&lt;h3 id=&quot;results-7&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Datasets&lt;/strong&gt;:  Babi&lt;/li&gt;
  &lt;li&gt;M2MN2N performs almost as good as the supervised M2M. More hops help the model&lt;/li&gt;
  &lt;li&gt;At each hop, the matrix A and C are shared.&lt;/li&gt;
  &lt;li&gt;Join training on all tasks helps&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;adversarial-generation-of-natural-language&quot;&gt;Adversarial Generation of Natural Language&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1705.10929.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-10&quot;&gt;Problems&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;Training GAN for text generation is hard: in the context of text generation, the generator is asked to generate a fake sequence of tokens, which is then passed to the discriminator. Extracting the token from the generator often involved a nondifferentiable operation: arg max.&lt;/li&gt;
  &lt;li&gt;Generating text from noise is hard&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;solutions-8&quot;&gt;Solutions&lt;/h3&gt;
&lt;h4 id=&quot;problem-1-3&quot;&gt;Problem 1&lt;/h4&gt;
&lt;p&gt;Instead of computing the arg max in the generator, the generator stops at the probability distribution (i.e \(softmax(o_t^TM)\)), where \(o_t\) is the output of a RNN cell at time-step \(t\), and M a dictionary matrix. This distribution vector list is passed to the discriminator, which is also a recurrent neural network. For sentences coming from the true distribution, we pass one hot vector instead, with each vector serving as an indicator of the observed word in the sample. I supposed they maybe used some feature hashing. At the last time-step, the discriminator is a binary logistic regression, determining how likely this sequence comes from the true distribution. Different loss have been employed (classic GAN, WGAN, LSGAN).&lt;br /&gt;
During inference, they always take the argmax instead of softmax one.&lt;/p&gt;

&lt;h4 id=&quot;problem-2-3&quot;&gt;Problem 2&lt;/h4&gt;
&lt;p&gt;For the encoding of a fake sentence, at each time-step, the RNN receives as input some Gaussian noise. To create coherent sequences, classical RNN probabilistic model conditioned every output on the previous input. However, when we want to generate sample from noise, this conditioning does not help the model. For this reason, they added a peephole connection between the output \(o_t\) of the RNN and every gates \((i_{t+1}, o_{t+1}, f_{t+1})\) at the next time-step. I also think they could have used something like teacher forcing instead of always passing noise.&lt;/p&gt;
&lt;h3 id=&quot;results-8&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: Chinese poetry&lt;/li&gt;
  &lt;li&gt;Still not better than MLE approach, but on its way&lt;/li&gt;
  &lt;li&gt;Best results with WGAN with gradient penalty instead of weight clipping.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;a-hierarchical-recurrent-encoder-decoder-for-generative-context-aware-query-suggestion&quot;&gt;A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1507.02221.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-11&quot;&gt;Problems&lt;/h3&gt;
&lt;p&gt;Predicting the next query given the previous one. Model using query co-occurrence lacks the ability of generalization for unseens or rare query sequence.&lt;/p&gt;

&lt;h3 id=&quot;solutions-9&quot;&gt;Solutions&lt;/h3&gt;
&lt;p&gt;Three RNNs are used in the model.&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Query level encoder&lt;/strong&gt;: The first one encode a query at the word level. The Hidden state is initialized to zero. At the last time-step, it returns a vector representing the all sentence \(h_q^n\) where \(n\) is the total number of queries in the sequence.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Session level encoder&lt;/strong&gt;: This encoder is an RNN, which takes as input the query embedding.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Next query decoder&lt;/strong&gt;: This RNN generates the next query given the previous ones. Its initial states is a linear transformation of the output of the session encoder output. Instead of using the using the recurrent state directly to compute the probability distribution, they used a linear transformation taking into account the previous recurrent states, and the previous word embedding decoded.&lt;br /&gt;
Trainning is done by maximizing the log-likelihood. BPTT also is used.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Side Note&lt;/strong&gt;: This paper has been extended to dialogue in this article &lt;a href=&quot;https://arxiv.org/pdf/1507.04808.pdf&quot;&gt;Building End-to-End Dialogue Systems Using Generative Hierarchical Neural Network Models&lt;/a&gt;&lt;/p&gt;
&lt;h3 id=&quot;results-9&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: AOL search log dataset. 16M search for 600.000 different users&lt;/li&gt;
  &lt;li&gt;State of the art results&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;scheduled-sampling-for-sequence-prediction-with-recurrent-neural-networks&quot;&gt;Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1506.03099.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-12&quot;&gt;Problems&lt;/h3&gt;
&lt;p&gt;This paper tackles the problem of structured output prediction, in the specific case where the output is a sequence and we represent the sequence as a directed graphical model that generates from the first token to the last. The paper starts from the observation that training such models by maximum likelihood does not reflect well how the model is used at test time. Indeed, ML training implies that the model is effectively trained to predict each token conditioned on the previous tokens from the ground truth sequence (this is known as “teacher forcing”). When predicting a new input, the model will generate a sequence by generating tokens one after another and conditioning on its predicted tokens instead.&lt;/p&gt;

&lt;h3 id=&quot;solutions-10&quot;&gt;Solutions&lt;/h3&gt;
&lt;p&gt;A different training procedure is proposed, where at training time each conditioning ground truth token is sometimes replaced by the model’s previous prediction. The choice of replacing the ground truth by the model’s prediction is made by “flipping a coin” with some probability, independently for each token. Importantly, the authors propose to start with a high probability of using the ground truth (i.e. start close to ML) and anneal that probability closer to 0, according to some schedule (thus the name Schedule Sampling).&lt;br /&gt;
Scheduled Sampling helps is that ML training does not inform the model about the relative quality of the errors it can make. Regarding ML, it is as bad to put high probability on an output sequence that has just one token that’s wrong; then it is to put the same amount of probability on a sequence that has all symbols wrong.&lt;br /&gt;
There is still a problem; the current algorithm ignores the fact that, by sometimes taking as input its previous prediction, this induces an additional relationship between the model’s parameters and its ultimate prediction, a relationship that isn’t considered during training.&lt;/p&gt;

&lt;h1 id=&quot;convolutional-sequence-to-sequence-learning&quot;&gt;Convolutional Sequence to Sequence Learning&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/pdf/1705.03122.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2017&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-13&quot;&gt;Problems&lt;/h3&gt;
&lt;p&gt;Training RNN is hard, long, and nonparallelizable. CNN has been applied with success to a wide range of applications, but not to sequence to sequence modeling.&lt;/p&gt;
&lt;h3 id=&quot;solutions-an-overview-of-the-model-1&quot;&gt;Solutions… an overview of the model&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;Convolutional Neural Network does not depend on the computation of the previous time step&lt;/li&gt;
  &lt;li&gt;Multi convolutional neural network allows nearby input elements to interact together at the bottom layer, while long term dependencies interact at higher layer. Hierarchical structure provides a shorter path for long term dependencies. Fixing the number of known linearities also eases learning.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here is an overview of the model:&lt;br /&gt;
As input, the model receives a 2D array where the first dimension is the size of the sequence \(m\), and the 2nd dimension corresponds to the embedding size \(d\).  During decoding, the already computed output has the same shape. Furthermore, in each row is also added a positional embedding to give a sense to the model of which portion of the sequence in the input or output it is currently dealing with.&lt;br /&gt;
Convolution kernel \(W \in \mathbb{R}^{2d x kd}\), where \(k\)  is the kernel size are used to performs 1D convolution along the first axis. Hence \(k\) rows  are embedded into a single vector \(y\) of size \(\mathbb{R}^{2d}\). As we doubled the last dimension, we can use a gated linear unit as the nonlinearity (second half of the vector is used as a gated vector on the first half). Furthermore, to enable deep convolutional networks, they added residual connections from the input of each convolution to the output of the block.&lt;/p&gt;

&lt;p&gt;For the encoder network, the output of the convolutional layers matches the input by padding the input. For the decoder network, they padded the input by \(k - 1\) element son both the left and right side and removed \(k\) elements from the end of the convolution output.&lt;/p&gt;

&lt;p&gt;Furthermore, a linear mapping between the embedding size is done before the encoder input, at the output of the encoder, to the final layer of the decoder \(h^L\) before the softmax, and to all decoder layers \(h^l\) before computing attention scores.&lt;/p&gt;

&lt;p&gt;For each decoder layer, they introduced a separate &lt;strong&gt;attention mechanism&lt;/strong&gt;. At every layer \(l\), for every non linearity \(i\), the intermediate vector \(h_i^l\) is linearly combined with the embedding decoder input “at time-step” \(i\) (i.e \(h_i = W_d^l h_i^l + b_i + g_i\)). From this a probability distribution \(\alpha_{ij}\) is computed between \(h_i\) and all encoder output \(z_j\). This weights are used to compute a weighting sum for every intermediate vector as \(c_i =  \sum_{j=1}^m \alpha_{ij}^l (z_j +  e_j)\) where \(e_j\) is the embedding for the \(j\)th input. It is a more  complex attention mechanism than in seq2seq model because at every decoder layer, for every intermediate vector, an conditionning is done over the encoder input, encoder output, and the previous decoder input. Once calculated, the attention vector is added to the corresponding decoder layer.&lt;/p&gt;

&lt;p&gt;A lot of normalization: residual blocks are normalized to keep the variance unchanged.  More precisely, to keep variance bounded when forwarding, they initialized weight matrices with a gaussian \(\mathbb{N}(0, \sqrt(\frac{4}{kernelsize}))\)&lt;/p&gt;

&lt;h3 id=&quot;results-10&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: WMT English-Romanian, English-French, English-German&lt;/li&gt;
  &lt;li&gt;Slightly better than RNN, but model was highly optimized with extensive HP search&lt;/li&gt;
  &lt;li&gt;From 7 to 21 faster on GPU/CPU.&lt;/li&gt;
  &lt;li&gt;Removing the source position embeddings results in a larger accuracy decrease than target position embeddings.&lt;/li&gt;
  &lt;li&gt;2 layers decoder performs good, at nine layers there is no more gain in accuracy.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;coherent-dialogue-with-attention-based-language-models&quot;&gt;Coherent Dialogue with Attention-based Language Models&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://arxiv.org/abs/1611.06997&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2017&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-14&quot;&gt;Problems&lt;/h3&gt;
&lt;ol&gt;
  &lt;li&gt;seq2seq models are better suited for converting the same information from one modality to another, than to dialogue modeling&lt;/li&gt;
  &lt;li&gt;Current attention mechanism supports attention over the output of the encoder, what if we extended to all the previous output of the generated sentence.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&quot;solutions-11&quot;&gt;Solutions&lt;/h3&gt;
&lt;h4 id=&quot;problem-1-4&quot;&gt;Problem 1&lt;/h4&gt;
&lt;p&gt;Their architecture (RNN Language Model) is a simple recurrent neural network which takes as input a context + question and predicts the next utterance. Model is trained via Maximum Likelihood, (teacher forcing)&lt;/p&gt;

&lt;h4 id=&quot;problem-2-4&quot;&gt;Problem 2&lt;/h4&gt;
&lt;p&gt;They created a new attention mechanism model (A-RNN). Attention vectors at time-step t are formulated combination of all the hidden vector \(h_{i \lt t}\) previous time-step, and the input embedding vector \(r_{i \lt t}\), such that \(\beta_{ti} = b^t tanh(W h_{it + U r_{it}}\), then applied a softmax to get a correct probability distribution, and finally compute attention as a convex sum \(z_t = \sum_{i=1}^t \alpha_{it}r_{it}\). This attention vector is used during the prediction of the token&lt;/p&gt;

&lt;h3 id=&quot;results-11&quot;&gt;Results&lt;/h3&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Dataset&lt;/strong&gt;: MovieTriples and Ubuntu TroubleShoot&lt;/li&gt;
  &lt;li&gt;On MovieTriples, smallest perplexity compared to rnn, seq2seq, and HRED&lt;/li&gt;
  &lt;li&gt;Compared to classical rnn language modeling, recall@1, recall@2 is better (~+5%).&lt;/li&gt;
  &lt;li&gt;Measure of diversity distinct-1 is better than traditional rnn without dynamic attention&lt;/li&gt;
  &lt;li&gt;On Human Evaluation, A-RNN is 90% at least better than simple RNN&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;decoupled-neural-interfaces-using-synthetic-gradients&quot;&gt;Decoupled Neural Interfaces Using Synthetic Gradients&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;:  &lt;a href=&quot;https://arxiv.org/pdf/1608.05343.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2016&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-15&quot;&gt;Problems&lt;/h3&gt;
&lt;p&gt;Training Neural Networks is sequential in three ways: each layer has to wait for the previous layer to do some computation, each layer has to wait for future layer to give back some information about the backpropagated signal, and also backpropagation can occur only afterward a forward pass has been done before&lt;/p&gt;

&lt;h3 id=&quot;solutions-12&quot;&gt;Solutions&lt;/h3&gt;
&lt;p&gt;Every layer becomes a sort of independent module, and it updates its weights based on gradient computed by a neural network at the next layer. This neural network receives as input the post activation \(o_l\) of layer \(l\), and predicts what would be the partial derivative \(\frac{dL}{do_l}\). Papers mentioned that the input of the gradient predictor could contain other information that the output weight, but it appeared not significantly to improve the model. &lt;br /&gt;
Only the last layer is trained from the data.&lt;br /&gt;
The whole point of this technique is to allow individual layer to train without waiting on each other to finish forward and backpropagating. You train a layer based on the gradient of the Synthetic Gradient module of layer \(l+1\), and when error from loss is finally backpropagated, you updated the SG module, and pass the gradient to the previous layer&lt;/p&gt;

&lt;h1 id=&quot;hybrid-computing-using-a-neural-network-with-dynamic-external-memory&quot;&gt;Hybrid computing using a neural network with dynamic external memory&lt;/h1&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Paper&lt;/strong&gt;: &lt;a href=&quot;https://www.gwern.net/docs/2016-graves.pdf&quot;&gt;–&amp;gt; * &amp;lt;–&lt;/a&gt;, 2016&lt;/li&gt;
&lt;/ul&gt;

&lt;h3 id=&quot;problems-16&quot;&gt;Problems&lt;/h3&gt;
&lt;p&gt;Neural network model lacks the ability to access to memory. Previous architectures use external memory. DNC try to remedy this issue.&lt;/p&gt;

&lt;h3 id=&quot;solutions--an-overview-of-the-model&quot;&gt;Solutions … an overview of the model&lt;/h3&gt;
&lt;p&gt;DNC uses a differentiable attention mechanism to define distribution over the N rows of the memory (size \(N x W\))
&lt;strong&gt;Reading and Writing&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;\(r\) is a read vector, and \(r = \sum_{i=1}^N M[i, *]w^r(i)\), where \(w^r\) is defined as a weighting vector&lt;/li&gt;
  &lt;li&gt;\(v\) is a write vector, it also used a weighting \(w^w\) with it. Writing is done for a column-row value in the memory (i.e \(M[i, j] = M[i, j] (1 - w^w[i] e[j]) +  w^w[i] v[j] \))&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There are three types of interaction between heads and memory.&lt;/p&gt;
&lt;ol&gt;
  &lt;li&gt;Content lookup between a key vector emitted by the controller and all the locations in memory. The similarity score defined a weighting used by the read head for associative recall, or by the writing heads for modifying the existing vector in memory.&lt;/li&gt;
  &lt;li&gt;Records transitions between consecutively written location in a \(N x N\) matrix. \(L(i, j)\)  is closed to 1 if I have been written after j.&lt;/li&gt;
  &lt;li&gt;The usage of each location is represented as a number between 0 and 1, and a weighting picks unused areas is delivered to the write head. It is automatically increased, decreased, after every write, respectively read.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;The Controller&lt;/strong&gt;&lt;br /&gt;
The controller network \(\mathbb{N}\) receives at each time-step \(t\) a vector \(x_t\), and emits an output vector \(y_t\), that could parametrizes a predictive distribution for a target vector \(v_t\). \(\mathbb{N}\) also recieves a set of R read vectors from the memory matrix at the previous time-step via the read heads. It then emits an \(\epsilon\) vector that defines its interactions with the memory at the current time-step. The controller network could be a LSTM cell for example, and all inputs (vector + read vectors) are concatenated.&lt;br /&gt;
\(v_t\) and \(\epsilon\) are functions of previous hidden vector, and \(y_t =  v_t + W[r_t^1, …, r_t^R]\).&lt;/p&gt;

&lt;p&gt;Before being used to parametrized memory interactions, \(\epsilon_t\) vector is subdivided into multiple vectors:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;R read keys (in \(\mathbb{R}^W\))&lt;/li&gt;
  &lt;li&gt;R read strenghts (in \([1, \inf)\))&lt;/li&gt;
  &lt;li&gt;A write key (in \(\mathbb{R}^W\))&lt;/li&gt;
  &lt;li&gt;A write strengh (***)&lt;/li&gt;
  &lt;li&gt;An erase vector (in \([0, 1]^W\))&lt;/li&gt;
  &lt;li&gt;A write vector (in \(\mathbb{R}^W\))&lt;/li&gt;
  &lt;li&gt;R free gates (in \([0, 1]\))&lt;/li&gt;
  &lt;li&gt;An allocation gate (in \([0, 1]\))&lt;/li&gt;
  &lt;li&gt;A write gate (in \([0, 1]\))&lt;/li&gt;
  &lt;li&gt;R read modes (in \([0, 1]\))&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Memory addressing&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;strong&gt;Content based addressing&lt;/strong&gt;
\(C(M, k, \beta)[i] = \frac{exp(cossim(k, M[i,]) \beta)}{sum_{j=1}^Nexp(cossim(k, M[j,]) \beta}\), where \(k \in \mathbb{R}^W\) is a lookup key, and \(\beta \in [1, \inf)\) is the key strengh.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;strong&gt;Dynamic memory allocation&lt;/strong&gt;
We need to define a memory usage vector at time \(t\) (in \([0, 1]^N\), to do so we will used the free gates \(f_t^i\) outputed by the controller and define the memory retention vector \(\sigma_t \in [0, 1]^N\) which represents by how each location will not freed by the free gates \(\sigma_t = \prod_{i=1}^R(1 - f_t^i w_{t-1}^{r, i})\). If a row is read from the memory at the previous time-step, then highest is a read weighting, highest it will becomes a free location for writing (if the free gates has decided so). Because,free gates allows the neural network to decides if a row should be kept longer in memory even thought it has already been read. Afterwards, the usage vector is defined as \(u_t = (u_{t-1} + w_{t-1}^w - u_{t-1} \cdot w_{t-1}^w) \cdot \sigma_t\). Once \(u_t\) has been determined, we sort the vector and return a list of the ascending sorted index. First rows are the least used memory slot. Finally they defined an allocation weighting equals to \(a_t[\sigma_t[j]] = (1 - u_t[\sigma_t[j]]) \prod_{i=1}^{j-1}u_t[\sigma_t[i]]\). If all usages are one, then the controller cannot allocate memory without first freeing used locations.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Temporal memory linkage&lt;/strong&gt;
We first need a precedence weighting \(p_t \in [0, 1]^N\) where \(p_t[i]\) is the degree to which location i was the last one written to: \(p_t = (1 - \sum_{i=1}^N w_t^w[i])p_{t-1} + w_t^w\). Every time a location is modified, the link matrix is updated to remove odd links to and from that location:&lt;/li&gt;
  &lt;li&gt;\(L_0(i, j) = 0\text{ } \forall {i, j}\)&lt;/li&gt;
  &lt;li&gt;\(L_t(i, i) = 0\text{ } \forall i\)&lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;\(L_t(i, j) = (1 - w_t^w[i] - w_t^w[j])L_{t-1}[i, j] + w_t^w[i] p_{t-1}[j]\)
From this matrix are created two \(* R\) vectors: \(fo_{t}^i = L_t w_{t-1}^{r, i}\), and \(b_t^i = L_t^T w_{t-1}^{r, i}\). Every vector is of dimension \(N\) and represents to which degree a row  \(j\) (i.e entry at index \(j\)), should be read whether because previously we read a row written just before, or we  read a row written just after.&lt;br /&gt;
All of this should be done after computing the allocated weights for write.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;&lt;strong&gt;To determine where to write in memory&lt;/strong&gt;
The system uses a combination of content based addressing and dynamic memory allocation. The controller can write to new allocated locations, or to location based on content addressing. First a write content weighting \(c_t = W(M_{t-1}, k_t^w, \beta_t^w)\) is constructed with the write keys, and the write strength. This weighting is interpolated with the allocated weights (i.e \(w_t^w = g_t^w[g_t^a a_t + (1 - g_t^a) c_t^w\)) to determined a writing weighting \(w_t^w\), where \(g_t^*\) are the write and allocated gates. If the write gates are zero, then nothing is written independently of other parameters.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;To determine where to read&lt;/strong&gt;
The model uses a combination of content based addressing and temporal memory linkage. First a read weighting for each read head is created \(c_t^{r, i} = W(M_{t-1}, k_t^{r, i}, \beta_t^{r, i})\). Each read head also receives a read mode vector \(\pi \in [0, 1]^3\), which interpolates among the backward weighting, the forward weighting, and the content read weighting, hence determining the read weighting: \(w_t^{r, i} = \pi[0] b_t^i + \pi_t^i[1]c_t^{r, i} + \pi_t^i[2]f_t^i\). If \(\pi[1]\) dominates the read mode, then the weighting reverts to content lookup, otherwise \(\pi[0], pi[2]\) dominates, then the read head iterates in reverse order, forward order respectively.&lt;/li&gt;
&lt;/ul&gt;

&lt;!-- Not well understood
# A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
* __Paper__:  * ](https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14567/14219)
* _Problem__:
    1. RNN often follows a flat generation process where the model variability only occurs at the output level, where we decided which word will be used (i.e., \\(p(w_t\|w_{1, ...t-1})\\) where \\(w_t\\)  is the generated token at time-step \\(t\\). Natural language has at least two level of structure: at a word level where the structure is dominated by word co-occurrence, and at a higher level across utterance where dialogue topic and speaker goals are expressed. If stochasticity is employed only at the output word level, then the model will have to decide on the conversation topic incrementally as it is generating words. This leads to incoherent topics and user goals.
    2. In classical seq2seq model, the decoder hidden state has to remember all of the previous contexts to generate the next token, but because of vanishing gradient, the short term goals will dominate the output distribution. 
###  Solutions 
    1. Add stocasticity before generating an utterance. Architecture is similar to the HRED. The hidden vector of the session level encoder is passed throught two feedforward neural network to output mean, and diagonal covariance of the approximate posterior distribution \\(Q  \simeq p(z\|w_1, ..., w_N)\\). The latent variable are inferred by maximizing the variational bound of the joint probability of all utterances \\(p(w_1, w_2, ..., w_N\\), which factorizes into independant terms:
    * \\(\sum_i^N -KL(Q(z_i\| w_1, ..., w_i) \|\| p(z_i\|w_{\lt i}) )\\) where \\(Q\\) will be a Gaussian
    * \\(\mathrm{E}_{z_n \sim Q}(p(w_n\| w_1, ..., w_{n-1}, z_n))\\) --&gt;
</description>
        <pubDate>Fri, 02 Jun 2017 21:39:00 -0400</pubDate>
        <link>http://louishenrifranc.github.io/mathblog/2017/06/03/trueai/</link>
        <guid isPermaLink="true">http://louishenrifranc.github.io/mathblog/2017/06/03/trueai/</guid>
        
        <category>math</category>
        
        <category>NLP</category>
        
        <category>deep learning</category>
        
        
        <category>mathblog</category>
        
      </item>
    
      <item>
        <title>The Hyperband Algorithm.</title>
        <description>&lt;p&gt;In this article, we will review a recent approach to hyper parameter search.&lt;/p&gt;

&lt;h2 id=&quot;why-a-new-hyper-parameter-search-algorithm&quot;&gt;Why a new hyper parameter search algorithm?&lt;/h2&gt;
&lt;p&gt;In the past month, I’ve been spending some time on Bayesian optimization, and they’ve become my favorite way to go for hyper parameter search. I’ve done a simple module for using Spearming software (an implementation of Bayesian optimization) in the context of deep learning using &lt;a href=&quot;https://github.com/louishenrifranc/spearmint-for-neural-network&quot;&gt;Lasagne Library&lt;/a&gt;. In the context of hyper parameter optimization, I also tried a reinforcement approach following &lt;href link=&quot;https://arxiv.org/pdf/1611.02167.pdf&quot;&gt;Designing Neural Network Architecture using Reinforcement Learning&lt;/href&gt;, without success (see &lt;href link=&quot;https://github.com/louishenrifranc/INF8225/blob/master/TP3/codes/QTable.py&quot;&gt;code&lt;/href&gt; . One of my claims is that it required way too much computational resource that many machine learning researcher can’t afford. At the same point, even with good GPUs, it was time-consuming, because it required training neural network for some epoch ruled by early stopping, and I realized that most of the training configuration where not promising since the beginning. The only way to avoid this nonpromising expansion was to whether trained only promising neural network architecture, or hard-coded rules in early stopping method for nonpromising training.&lt;/p&gt;

&lt;p&gt;Instead, in HYPERBAND, they explore a different direction for hyper parameter optimization that focuses on speeding up configuration evaluation. I decided it was the moment to try it. By adaptively allocating these resources, their approach aims to examine orders of magnitude more hyper parameter configurations than approaches that uniformly train all configurations to completion, thereby quickly identifying good hyperparameters.&lt;/p&gt;

&lt;p&gt;First, they proved that Bayesian optimization (SMAC and TSE) consistently outperforms random sampling, but by a small gap. However, given more capacity, let’s say two computer instead of one, random search always produces better results.&lt;/p&gt;

&lt;p&gt;&lt;img style=&quot;float: center;&quot; src=&quot;/img/2017-05-08-hyperband/rank_chart.png&quot; width=&quot;600&quot; /&gt;&lt;/p&gt;

&lt;p&gt;This is their primary motivation for improving random search, by allocating different amount of resources for particular hyper parameter configuration.&lt;/p&gt;

&lt;h3 id=&quot;successivehalving&quot;&gt;SuccessiveHalving&lt;/h3&gt;
&lt;p&gt;HYPERBAND is an extension of the SuccessiveHalving algorithm. Here is a simple pseudo-python code of the last one.&lt;/p&gt;

&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;n&quot;&gt;budget&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1000&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# time or nb_epochs to allocate for hyper parameter search &lt;/span&gt;
              &lt;span class=&quot;c&quot;&gt;# Note that this we can do extend search without exploding the budget&lt;/span&gt;
              &lt;span class=&quot;c&quot;&gt;# by parallelizing tasks&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;nb_configuration&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;n&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;30&lt;/span&gt; &lt;span class=&quot;c&quot;&gt;# number of configuration to explore&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;all_configurations&lt;/span&gt;  &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;get_new_configuration&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;_&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;range&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nb_configuration&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)]&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;while&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;all_configurations&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;!=&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;c&quot;&gt;# Save current proxy score for configuration selection &lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;score_configuration&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{}&lt;/span&gt;
    &lt;span class=&quot;c&quot;&gt;# Saved currently trained model&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;saved_model&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{}&lt;/span&gt;
    
    &lt;span class=&quot;c&quot;&gt;# Reduce budget&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;budget&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;  &lt;span class=&quot;n&quot;&gt;budget&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;all_configurations&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;log&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;nb_configuration&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;log&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)))&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;configuration&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;all_configurations&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;c&quot;&gt;# Create a new model the first time,&lt;/span&gt;
        &lt;span class=&quot;c&quot;&gt;# then restore the model&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;configuration&lt;/span&gt; &lt;span class=&quot;ow&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;saved_model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;saved_model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;configuration&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
        &lt;span class=&quot;k&quot;&gt;else&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;
            &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;create_new_model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;hyperparameter&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;configuration&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
        
        &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;train_algorithm&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;until_epoch&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;last_epoch&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;+&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;budget&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;/&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;all_configurations&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;
        
        &lt;span class=&quot;n&quot;&gt;saved_model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;configuration&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;
        &lt;span class=&quot;n&quot;&gt;score_configuration&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;model&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;get_validation_score&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;()&lt;/span&gt;

    &lt;span class=&quot;c&quot;&gt;# Sort model&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;index_sorted&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;arg_sort&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;score_configuration&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;all_configurations&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;all_configurations&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;index_sorted&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;

    &lt;span class=&quot;c&quot;&gt;# Save half of the model for next round&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;saved_model&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;saved_model&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;[:&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;len&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;all_configurations&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&lt;/span&gt; &lt;span class=&quot;mi&quot;&gt;2&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;hr /&gt;

&lt;h4 id=&quot;notes&quot;&gt;Notes&lt;/h4&gt;
&lt;p&gt;It is easy to show that the allocated budget for the algorithm does not exceed the &lt;code class=&quot;highlighter-rouge&quot;&gt;budget&lt;/code&gt;:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\text{All budget allocated = }\displaystyle{\sum_{k=0}^{log_2(n) - 1} \lfloor \|P_k\| \frac{budget}{|P_k|\lceil log_2(n)\rceil)\rfloor}}&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\text{                     }\displaystyle{ \leq \sum_{k=1}^{log_2(n)} - 1}\frac{budget}{floor(log_2(n)))}&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\text{                     }\displaystyle{ \leq budget}&lt;/script&gt;

&lt;p&gt;where \(P_k =\) &lt;code class=&quot;highlighter-rouge&quot;&gt;len(all_configurations)&lt;/code&gt;, and \(k\) is the current iteration of the algorithm.&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;In practice, the optimal allocation strategy is unknown because we do not have knowledge of the envelope functions nor the distribution of terminal losses. Hence at the beginning, it may be unfair to differentiate model based uniquely on the first iteration validation curves. A simple example is to compare gradient descent based model with low learning rate: they usually perform badly at first before converge slowly to a better solution. &lt;strong&gt;SuccessiveHalving&lt;/strong&gt; will usually throw them at the first step, without further consideration. Not desired.&lt;/p&gt;

&lt;h2 id=&quot;pseudo-code&quot;&gt;Pseudo Code&lt;/h2&gt;
&lt;p&gt;There are two parameters to set for the algorithm&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;\(R\)&lt;/strong&gt;: The maximum number of time/epoch to allocate to a single configuration. I think this is more relevant than having to set a global budget. In the context of Machine Learning problem we often set a fixed number of epochs and used upon some early stopping technique.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;\(\eta\)&lt;/strong&gt;: The proportion of configuration to discard at each iteration.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now here is the algorithm:
&lt;img style=&quot;float: center;&quot; src=&quot;/img/2017-05-08-hyperband/hyperband_pseudo_code.PNG&quot; width=&quot;600&quot; /&gt;&lt;/p&gt;

&lt;center&gt;&lt;div style=&quot;float: center;&quot; id=&quot;visualization1&quot;&gt;&lt;/div&gt;&lt;/center&gt;
&lt;p&gt;&lt;em&gt;&lt;center&gt;&lt;strong&gt;r&lt;/strong&gt; as a function of R and s&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;

&lt;center&gt;&lt;div style=&quot;float: center;&quot; id=&quot;visualization2&quot;&gt;&lt;/div&gt;&lt;/center&gt;
&lt;p&gt;&lt;em&gt;&lt;center&gt;&lt;strong&gt;n&lt;/strong&gt; as a function of R and s&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;There are two loops in the algorithm.&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;The first outer loop computed (\(n\) and \(r\)).&lt;/li&gt;
  &lt;li&gt;The second inner loop computed a full iteration of a &lt;em&gt;SuccessiveHalving&lt;/em&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At the beginning, \(n\) is set large for early value of \(s\), and it’s the opposite for \(r\). Hence in the inner loop, it will allow exploring a lot of configuration a few iteration very quickly.
However as \(s\) decreased (outer loop), \(r\) is set larger, but n decreased. hence fewer model will be tried, &lt;strong&gt;but&lt;/strong&gt; those will be tried longer. &lt;strong&gt;It will allow model that performed great lately to be revealed…&lt;/strong&gt;.&lt;/p&gt;

&lt;!-- 
## Implementation
I decided to implement Tensorflow model, trying it on CIFAR10, (because I'm tired of MNIST), but the interface should be reusable for any model. Our API should accept any model that had functions to:
* Train for X epoch,
* (Re)Store trained model,
* Return a validation metric

We also need a random generator for hyperparameter configuration.  
Finally, it would also be nice to have search results in readable forms, like an array. --&gt;
&lt;!-- The current implementation is![here](https://github.com/louishenrifranc/hyperband). The project is not over. --&gt;

&lt;h2 id=&quot;bibliography&quot;&gt;Bibliography&lt;/h2&gt;
</description>
        <pubDate>Sun, 07 May 2017 21:39:00 -0400</pubDate>
        <link>http://louishenrifranc.github.io/mathblog/2017/05/08/hyperband/</link>
        <guid isPermaLink="true">http://louishenrifranc.github.io/mathblog/2017/05/08/hyperband/</guid>
        
        <category>math</category>
        
        <category>machine-learning</category>
        
        <category>r</category>
        
        <category>statistic</category>
        
        
        <category>mathblog</category>
        
      </item>
    
      <item>
        <title>Image Generation with BEGAN and Skip-Though vector.</title>
        <description>&lt;h2 id=&quot;skip-thought-vector&quot;&gt;Skip Thought Vector&lt;/h2&gt;
&lt;p&gt;Skip thought vector is a powerful technique to map semantically and similar syntactic sentences to same vector representation. Since 2015, the paper has already been cited more than &lt;em&gt;250&lt;/em&gt; times, which is entirely deserved.&lt;br /&gt;
Until 2015, the traditional deep learning ways to compute sentence embedding was to learn composition operators that maps (word2vec) word embeddings to sentence representations using diverse architecture such as recurrent network &lt;sup id=&quot;fnref:Schimi&quot;&gt;&lt;a href=&quot;#fn:Schimi&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;, recursive network &lt;sup id=&quot;fnref:Chung2014GRUvsLSTM&quot;&gt;&lt;a href=&quot;#fn:Chung2014GRUvsLSTM&quot; class=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;, and also convolution neural network &lt;sup id=&quot;fnref:Cnnsentences&quot;&gt;&lt;a href=&quot;#fn:Cnnsentences&quot; class=&quot;footnote&quot;&gt;3&lt;/a&gt;&lt;/sup&gt;. All these models fall under the supervision tasks; they required a loss function at the bottom of the neural network to be trained. Hence they are biased towards their respective functions.&lt;br /&gt;
Skip though vector is completely unsupervised, and is often compared to word2vec but at a sentence level.&lt;/p&gt;

&lt;p&gt;In essence, what skip thought vector is like the continuous skip-gram architecture of word2vec, but here a sentence is used to predict its surrounding sentences. 
Training data is in form of current sentence \(s_i\), the next sentence \(s_{i+1}\) and the previous sentence \(s_{i-1}\).
The current sentence is encoded at the word level, using a shared word2vec dictionnary between all encoder/decoders, and a classical GRU cell:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/img/2017-05-01-began/gru.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;At the end, the last output vector \(h_i\) is retrieved and will be used for predicting the surronding sentences. For this tasks, two decoders are shared which do not shared weights. Instead of a vanilla GRU cell they introduced three matrices &lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;\(C_r\), \(C_h\), \(C_z\)&lt;/span&gt; to bias the update, reset gates and the hidden states. Here are the formula. During decoding, at timestep \(t\), we have:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;\(r^t = \sigma(W_r^{dec} x^{t-1} + U_r^{dec} h^{t-1} + \) &lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;\(C_r\)&lt;/span&gt; \(h_i)\)&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;\(z^t = \sigma(W_z^{dec} x^{t-1} + U_z^{dec} h^{t-1} + \) &lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;\(C_z\)&lt;/span&gt; \(h_i)\)&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;\(h^t = tanh(W_h^{dec} x^{t-1} + U_h^{dec} h^{t-1} + \) &lt;span style=&quot;color:red; font-weight: bold;&quot;&gt;\(C_h\)&lt;/span&gt; \(h_i)\)&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As usual seq2seq models, the probability of having the word \(w_t^{i+1} = w_k\) in the next sentence at timestep \(t\) is computed using a softmax (or a sampling softmax), and a learnable dictionnary matrix for each words.&lt;/p&gt;

&lt;p&gt;That’s the theory, now how does it perform on &lt;a href=&quot;http://mscoco.org/&quot;&gt;MS Coco&lt;/a&gt; caption dataset.
Here is some representation in 3D of the vector obtained for every caption. I used &lt;em&gt;t-sne&lt;/em&gt; for the dimensionality reduction.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;left&gt;Figures 1: Closest caption to &quot;A cat laying on top of a computer keyboard&quot;&lt;/left&gt;&lt;/em&gt;
&lt;img style=&quot;float: right;&quot; src=&quot;/img/2017-05-01-began/computer_embedding.PNG&quot; width=&quot;200&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img style=&quot;float: left;&quot; src=&quot;/img/2017-05-01-began/red_bus.PNG&quot; width=&quot;300&quot; height=&quot;400&quot; /&gt;
&lt;em&gt;&lt;right&gt;Figure 2: Visualisation of the manifold of bus&lt;/right&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I have a lot of other examples, but this proves that skip-thoughts vector has captured some syntactic and semantics about captions. We’ll use them to try to generate images with the next architecture: a BEGAN.&lt;/p&gt;

&lt;h2 id=&quot;boundary-equilibrium-generative-adversarial-networks&quot;&gt;Boundary Equilibrium Generative Adversarial Networks&lt;/h2&gt;
&lt;h3 id=&quot;introduction-to-generative-adversarial-models&quot;&gt;Introduction to Generative Adversarial Models&lt;/h3&gt;
&lt;p&gt;GAN, GAN, GAN, there has been a huge craze for them because they were able to produce some of the nicest looking samples. This graphic resumes all about the success of the Generative Adversarial Model.
&lt;img src=&quot;/img/2017-05-01-began/cumulative_number_gan.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Generative Adversarial Model &lt;sup id=&quot;fnref:goodfellow&quot;&gt;&lt;a href=&quot;#fn:goodfellow&quot; class=&quot;footnote&quot;&gt;4&lt;/a&gt;&lt;/sup&gt; are model that are able to learn a data distribution and allows to sample from it.  The vanilla GAN objectives can be thought of as a minimax game played between the Generator (\(G\)) and the Discriminator (\(D\)) with the dueling objectives : \(\min_{G}\max_{D} V(D,G)\), where:&lt;/p&gt;

&lt;p&gt;\(V(D,G) = E_{x \sim p_{data}(x)}[\log D(x)] + E_{z \sim p_{z}(z)}[\log(1-D(G(z)))]\)&lt;/p&gt;

&lt;h4 id=&quot;current-issue-with-gans&quot;&gt;Current issue with GANS&lt;/h4&gt;
&lt;h5 id=&quot;unstable-training&quot;&gt;Unstable training&lt;/h5&gt;
&lt;p&gt;GANs training is giving a hard time to train because, defined as a zero-sum game, if one player becomes too powerful the “game is over,” and gradients could vanished before reaching the Generator.&lt;/p&gt;

&lt;h5 id=&quot;mode-collapse&quot;&gt;Mode collapse&lt;/h5&gt;
&lt;p&gt;Another big issue of GANs is mode collapse. The intuition for this is to look at another formula derived from the minimax equation. Finding the optimal \(p_G(z)\) among a class of generative models corresponds actually to minimizing the Jensen-Shannon:
&lt;script type=&quot;math/tex&quot;&gt;JS(P\|\|Q) = \displaystyle{\frac{1}{2}KL(P \|\|M) + \frac{1}{2}KL(Q\|\|M)  }&lt;/script&gt;&lt;/p&gt;

&lt;p&gt;where&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;\(M\) is a mixture of \(P\) and \(Q\): \(M = \frac{1}{2}(P + Q)\)&lt;/li&gt;
  &lt;li&gt;and KL is the Kullback Leibler divergence, defined in a continous space as \(\int_{-\infty}^{\infty} q(x) log(\frac{q(x)}{p(x)}) \, \mathrm{d}\mu_x\).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;How does the KL and the reverse KL divergence behaves?
&lt;img src=&quot;/img/2017-05-01-began/ml_vs_rkl.PNG&quot; alt=&quot;&quot; /&gt;
&lt;em&gt;&lt;center&gt;Figure 3: Approximated distribution given KL or reverse-KL training&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;In \(KL(P|Q)\), if \(P\) is multimodal, then the optimal \(Q\) will try to cover all modes, putting mass at point \(X\) of probability \(P(X) = 0\). It is because, in the formula of the \(KL\), \(Q\) is not penalized for putting mass on thoses points. This means that the generative model trained on KL divergence will tend to overgeneralize.&lt;/li&gt;
  &lt;li&gt;In \(KL(Q|P)\), the reverse KL divergence, \(Q\)  “leads” the training, so wherever \(Q\) doesn’t put any mass, it is not penalized. As a result, the optimal \(Q\)  is often at the largest mode of \(P\), ignoring other modes. We can see the dynamic of the training: as soon as the approximated distribution puts correct mass on a mode of \(p\), it will then try to fit perfectly the curve at that point, avoiding introducing unplausible samples, at the cost of missing other plausible samples. Training might get stuck producing only blue estate car.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So we could think because GANs is a combination of both KL, training them will behave halfway between the two extreme behaviors. Regrettably, because the reverse KL is super hard to optimize, and not well defined wherever \(P\) has a low dimensional support, this statement is false.&lt;/p&gt;

&lt;h5 id=&quot;no-measure-of-convergence&quot;&gt;No measure of convergence&lt;/h5&gt;
&lt;p&gt;Finally, with vanilla GANs, there is no way to measure convergence of the training, because the training loss doesn’t mean anything. A recent paper, WGAN &lt;sup id=&quot;fnref:wgan&quot;&gt;&lt;a href=&quot;#fn:wgan&quot; class=&quot;footnote&quot;&gt;5&lt;/a&gt;&lt;/sup&gt;, tackles this issue by introducing a new loss function, which is showed to correlate well with the perceptual quality of the sample generated by \(G\).&lt;/p&gt;

&lt;p&gt;The new loss function introduced to use the Earth Moving distance defined as:&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;“Probability distributions are defined by how much mass they put on each point. Imagine we started with distribution \(Pr\)​​, and wanted to move mass around to change the distribution into \(Pg\). Moving mass m by distance add costs \(m \cdot d\) effort. The earth mover distance is the minimal effort we need to spend.” &lt;a href=&quot;http://www.alexirpan.com/2017/02/22/wasserstein-gan.html&quot;&gt;From Read-through: Wasserstein GAN&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Another significant contribution of this paper is that it allows for training the Discriminator until optimality, and then used it to get “good” gradients for training the Generator. However, one drawback of this technique is that for the convergence property to hold, they had to enforce k-Lipschitzness for the function approximator (i.e., G and D). They decided to do it by clipping weights, which results in slow training and less capacity for the neural networks. In the next paragraph, we will see that BEGAN uses the different hypothesis to remove this constraint.&lt;/p&gt;

&lt;h3 id=&quot;began&quot;&gt;BEGAN&lt;/h3&gt;
&lt;h4 id=&quot;introduction&quot;&gt;Introduction&lt;/h4&gt;
&lt;p&gt;In essence, BEGAN uses an auto-encoder as a discriminator. With the auto encoder, BEGAN optimization tries to match the distribution of the errors instead of the usual used samples distribution.&lt;/p&gt;

&lt;p&gt;For training the auto encoder, they used a L1 loss (i.e \(L(v) =  |v - D(v)|^1\) where \(v\) is the input of the AE and \(D(v)\) the output).&lt;/p&gt;

&lt;p&gt;Given the hypothesis that the loss at the pixel level is independent and identically distributed (i.e \(\forall x_i\text{ a pixel, then }p(L(x_i), L(x_j)) = p(L(x_i)) p(L(x_j))\text{ }j \neq i, \), and all \(L(x_*)\) are generated from the same distribution, then by using the &lt;strong&gt;Central Law Theorem&lt;/strong&gt;, we can state, that as the number of pixels increased, the distribution can be approximated by a Gaussian distribution \(\mathcal{N}(\mu, \sigma)\).&lt;/p&gt;

&lt;p&gt;We can then find a closed form for the squarred Wassertein distance \(W\) between two Gaussians, and under certains constraints have a nice form of it:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;W(m_1 \propto \mathcal{N_1}, m_2 \propto \mathcal{N_2})^2 \propto \|\|\mu_1−\mu_2\|\|_2^2&lt;/script&gt;

&lt;p&gt;I think they started their demonstration with the squared Wasserstein distance \(W_2\) for Gaussian distribution formula because there exists an excellent established formula for it &lt;sup id=&quot;fnref:squarre_wdistance&quot;&gt;&lt;a href=&quot;#fn:squarre_wdistance&quot; class=&quot;footnote&quot;&gt;6&lt;/a&gt;&lt;/sup&gt;. See &lt;a href=&quot;https://stats.stackexchange.com/questions/83741/earth-movers-distance-emd-between-two-gaussians&quot;&gt;here&lt;/a&gt;. So the results holds for \(W_2\), but not for \(W_1\), which is nevertheless upper-bounded by \(W_2\). Now, their method probably minimizes \(W_1\) at the same time it would have minimized \(W_2\), but the connection is not cleared to my mind.
&lt;!-- TODO: Prove why we can get rid of the covariance terms. --&gt;&lt;/p&gt;

&lt;p&gt;So as we said earlier, we are going to assume our loss distribution is Normal and derived the math using this hypothesis. If we defined \(m_1\) as the auto encoder loss of the true image \(L(x)\) and \(m_2\) the loss  of the fake sample \(L(G(z))\), \(z\) being a random uniform latent vector.&lt;/p&gt;

&lt;p&gt;We can write the loss function as a GAN objective with two losses:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;L_D = L(x) - L(G(z)) \text{ for } \theta_D \text{ (weights of the auto encoder)}&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;L_D = L(G(z)) \text{ for } \theta_G \text{ (weights of the generator)}&lt;/script&gt;

&lt;p&gt;If you read the WGAN paper, both losses are quite similar, except that in BEGAN, it does not require any Lipschitzness constraints, because we assumed we had a Gaussian distribution, which gives us a different form of the Wasserstein distance.&lt;/p&gt;

&lt;p&gt;In Tensorflow this will be written as:&lt;/p&gt;
&lt;div class=&quot;language-python highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;k&quot;&gt;def&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;autoencoder_loss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;out&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;inp&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;eta&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;mi&quot;&gt;1&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;):&lt;/span&gt;
    &lt;span class=&quot;n&quot;&gt;diff&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;abs&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;out&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;inp&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;reduce_sum&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;tf&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;pow&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;diff&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;eta&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;))&lt;/span&gt;

&lt;span class=&quot;n&quot;&gt;mu_real&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;autoencoder_loss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;D_real_out&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;D_real_in&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;mu_gen&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;autoencoder_loss&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;(&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;D_gen_out&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;D_gen_in&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;)&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;D_loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mu_real&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;k_t&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mu_gen&lt;/span&gt;
&lt;span class=&quot;n&quot;&gt;G_loss&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;n&quot;&gt;mu_gen&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Did you notice the \(k_t\) in the code? Well, this has also been introduced in the BEGAN paper, and it will help ensure the stability of training.&lt;/p&gt;

&lt;h4 id=&quot;control-theory-and-convergence&quot;&gt;Control Theory and Convergence&lt;/h4&gt;
&lt;p&gt;In game theory, the Nash equilibrium is a solution concept of a non-cooperative game involving two or more players in which each player is assumed to know the equilibrium strategies of the other players, and no player has anything to gain by changing only his or her strategy. Similarly, in GANs, a possible equilibrium occurs when:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\mathop{\mathbb{E}}(L(x)) = \mathop{\mathbb{E}}(L(G(z)))&lt;/script&gt;

&lt;p&gt;If this equilibrium occurs, then G would have won over D, producing no distinguishable sample from the real distribution, and we don’t want this. We always want to maintain the two players, G, and D, “in the game.” To doing so, we will try to maintain this ratio to a specific value (Let’s call it \(\gamma = \frac{\mathop{\mathbb{E}}(L(G(z))}{\mathop{\mathbb{E}}(L(x))}\)) during training.&lt;/p&gt;

&lt;p&gt;Gamma will be an hyperparameter between \([0, 1]\), thus, and you’ll see soon, we wil enforce \(\mathop{\mathbb{E}}(L(x)) &amp;gt; \mathop{\mathbb{E}}(L(G(z)))\))&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;If \(\gamma\)  is close to 1, then the expected losses are similar, hence D is trying to improve at its auto-encoding tasks on both distribution, resulting in a more diverse output. &lt;!-- reformulate --&gt;&lt;/li&gt;
  &lt;li&gt;If \(\gamma\)  is small, then \(\mathop{\mathbb{E}}(L(x))\) becomes large, which means that the Discriminator is focusing more on improving its loss w.r.t to the true image. Diversity is impacted.
&lt;img style=&quot;float: center;&quot; src=&quot;/img/2017-05-01-began/lambda_variation.png&quot; /&gt;
&lt;em&gt;&lt;center&gt;Figure 4: Generated images given different gamma values&lt;/center&gt;&lt;/em&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;During training, we want to maintain the ratio of expectation to a fix value, which can be written as a equation constraint:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\gamma = \frac{\mathop{\mathbb{E}}(L(G(z))}{\mathop{\mathbb{E}}(L(x))}&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\gamma \mathop{\mathbb{E}}(L(x)) - \mathop{\mathbb{E}}(L(G(z)) = 0&lt;/script&gt;

&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Proportional_control#Proportional_Control_Theory&quot;&gt;Proportionnal Control Theory&lt;/a&gt; can do the job for us to adapt this ratio during optimization dynamically. The algorithm will update a factor \(k_t\), at each time step t, that will adapt depending on how far we are from the optimal desired ratio of expectation.&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;L_D = L(x) - k_t L(G(z))&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;L_D = L(G(z))&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;k_{t+1} = k_t + \lambda_k (\gamma L(x) - L(G(z))) \text{ for each timestep }t&lt;/script&gt;

&lt;p&gt;If \(k_t\) becomes too large (because \(L(x)\) is larger than \(L(G(z)))\), then the discriminator loss will be forced to focus more on \(L(G(z))\), and vice-versa, so none of the players can win the game.&lt;/p&gt;

&lt;p&gt;Training procedure also becomes simultaneous, while still being adversarial. There is no need to balance training period between them manually.&lt;/p&gt;

&lt;p&gt;Finally deriving a convergence measures becomes possible. We know we want to minimize the reconstruction loss on the correct image, and also maintain the equilibrium constant, thus, if the sum of this values \(L(x) + |\gamma L(x) - L(G(z))|\) gets low, then it is good proxy to say training converged.&lt;/p&gt;

&lt;h2 id=&quot;results&quot;&gt;Results&lt;/h2&gt;
&lt;h3 id=&quot;mscoco-with-conditionnal-began-with-gamma--055-norm-1&quot;&gt;MSCOCO with Conditionnal BEGAN with gamma = 0.55, norm 1&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;/img/2017-05-01-began/mscoco_055_1_nokl.gif&quot; width=&quot;200&quot; height=&quot;50&quot; /&gt;
&lt;em&gt;&lt;center&gt;Generated Images Evolution&lt;/center&gt;&lt;/em&gt;&lt;br /&gt;
&lt;br /&gt;
&lt;img src=&quot;/img/2017-05-01-began/mscoco_055_1_nokl_recon_false.gif&quot; width=&quot;200&quot; height=&quot;50&quot; /&gt;
&lt;em&gt;&lt;center&gt;Reconstruction of Generated Images Evolution&lt;/center&gt;&lt;/em&gt; 
&lt;br /&gt;
&lt;img src=&quot;/img/2017-05-01-began/mscoco_055_1_nokl_recon_true.gif&quot; width=&quot;200&quot; height=&quot;50&quot; /&gt;
&lt;em&gt;&lt;center&gt;Reconstruction of True Images Evolution&lt;/center&gt;&lt;/em&gt;&lt;br /&gt;
&lt;br /&gt;
&lt;img src=&quot;/img/2017-05-01-began/mscoco_055_1_nokl_true.gif&quot; width=&quot;200&quot; height=&quot;50&quot; /&gt;
&lt;em&gt;&lt;center&gt;True Images Evolution&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h3 id=&quot;mscoco-with-conditionnal-began-with-gamma--055-norm-2&quot;&gt;MSCOCO with Conditionnal BEGAN with gamma = 0.55, norm 2&lt;/h3&gt;
&lt;p&gt;&lt;img src=&quot;/img/2017-05-01-began/mscoco_07_2_nokl.gif&quot; width=&quot;200&quot; height=&quot;100&quot; /&gt;
&lt;em&gt;&lt;center&gt;Generated Images Evolution&lt;/center&gt;&lt;/em&gt;&lt;br /&gt;
&lt;br /&gt;
&lt;img src=&quot;/img/2017-05-01-began/mscoco_07_2_nokl_recon_true.gif&quot; width=&quot;200&quot; height=&quot;100&quot; /&gt;
&lt;em&gt;&lt;center&gt;Reconstruction of True Images Evolution&lt;/center&gt;&lt;/em&gt;&lt;br /&gt;
&lt;br /&gt;
&lt;img src=&quot;/img/2017-05-01-began/mscoco_07_2_nokl_true.gif&quot; width=&quot;200&quot; height=&quot;100&quot; /&gt;
&lt;em&gt;&lt;center&gt;True Images Evolution&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;To be fair, I didn’t expect great results. Original papers were trained on human faces; I trained it on a complex multimodal distribution, where different samples might share no similarities. It makes the job hard for the generator to pick which “type of image” to generate, hence resulting in artifacts. If I have more time, I’ll train BEGAN on LSUN dataset :)&lt;/p&gt;

&lt;h2 id=&quot;code&quot;&gt;Code&lt;/h2&gt;
&lt;p&gt;All the code for reproducing the experimentations is &lt;a href=&quot;https://github.com/louishenrifranc/began&quot;&gt;here&lt;/a&gt;&lt;/p&gt;

&lt;h2 id=&quot;footnotes&quot;&gt;Footnotes&lt;/h2&gt;
&lt;p&gt;&lt;a name=&quot;myfootnote1&quot;&gt;1&lt;/a&gt;:: This is mostly because when minimizing a \(L_2\) loss, this is the same as maximizing the log likelihood of a Gaussian distribution. However, images are usually complex distribution, with multiple modes. For example, there is not a single way to represent a car (it can be blue, yellow; it can be large, with or without a rooftop, and images are usually taken from a wide range of camera angle), thus doing ML can be interpreted as taking the average of all the representation of car to generate a new one in the manifold of automobiles, resulting in blurry images.
&lt;a name=&quot;myfootnote2&quot;&gt;2&lt;/a&gt;:&lt;/p&gt;

&lt;h2 id=&quot;bibliography&quot;&gt;Bibliography&lt;/h2&gt;

&lt;div class=&quot;footnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:Schimi&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;http://dl.acm.org/citation.cfm?id=1246450&quot;&gt;Long short-term memory&lt;/a&gt; from &lt;em&gt;Sepp Hochreiter and Jürgen Schmidhuber&lt;/em&gt;.&amp;nbsp;&lt;a href=&quot;#fnref:Schimi&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:Chung2014GRUvsLSTM&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf&quot;&gt;Recursive deep models for semantic compositionality over a sentiment treebank&lt;/a&gt; from &lt;em&gt;Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:Chung2014GRUvsLSTM&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:Cnnsentences&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;http://www.aclweb.org/anthology/P14-1062&quot;&gt;A convolutional neural network for modeling sentences&lt;/a&gt; from &lt;em&gt;Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom.&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:Cnnsentences&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:goodfellow&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1406.2661&quot;&gt;Generative Adversarial Networks&lt;/a&gt; from _ Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio_&amp;nbsp;&lt;a href=&quot;#fnref:goodfellow&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:wgan&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;https://arxiv.org/abs/1701.07875&quot;&gt;Wasserstein GAN&lt;/a&gt;  from &lt;em&gt;Martin Arjovsky, Soumith Chintala, Léon Bottou&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:wgan&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:squarre_wdistance&quot;&gt;
      &lt;p&gt;&lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/0047259X8290077X?np=y&quot;&gt;The Fréchet distance between multivariate normal distributions&lt;/a&gt; from &lt;em&gt;D.C Dowson, B.V Landau&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:squarre_wdistance&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</description>
        <pubDate>Sun, 30 Apr 2017 21:39:00 -0400</pubDate>
        <link>http://louishenrifranc.github.io/mathblog/2017/05/01/began/</link>
        <guid isPermaLink="true">http://louishenrifranc.github.io/mathblog/2017/05/01/began/</guid>
        
        <category>math</category>
        
        <category>machine-learning</category>
        
        <category>r</category>
        
        <category>statistic</category>
        
        
        <category>mathblog</category>
        
      </item>
    
      <item>
        <title>End-to-End Approach for making a Facebook bot using a Sequence to Sequence model.</title>
        <description>&lt;p&gt;If you arrived here from my blog, here is the link for the &lt;a href=&quot;https://github.com/louishenrifranc/facebook_bot&quot;&gt;code&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;As a fun school project with two folks, we decided to work on Sequence to Sequence model &lt;a href=&quot;/techblog/2017/04/21/seq2seq/&quot;&gt;see this blog&lt;/a&gt;, and we spent some time working to make a chatbot that will answer at our place on Facebook. For this, we used our own Facebook Messenger history (Technically some messages were Facebook chat message, I’m getting too old…).&lt;/p&gt;

&lt;p&gt;We train the model on a &lt;strong&gt;character level&lt;/strong&gt; because usually on Messenger, syntax, and grammar sucks. But if you have time, you could use this &lt;a href=&quot;https://github.com/gouwsmeister/TextCleanser&quot;&gt;soft&lt;/a&gt;. As our conversations were not in English, we didn’t use it, but it might worth and compare.&lt;/p&gt;

&lt;p&gt;For this project, you’ll need a Python interpreter (Make sure you installed all &lt;strong&gt;dependencies&lt;/strong&gt; &lt;code class=&quot;highlighter-rouge&quot;&gt;pip install -r requirements.txt&lt;/code&gt;), and a good GPU.&lt;/p&gt;

&lt;p&gt;Without any further ado, let’s go :)&lt;/p&gt;

&lt;h2 id=&quot;1-retrieve-and-parse-messenger&quot;&gt;1. Retrieve and parse Messenger&lt;/h2&gt;
&lt;p&gt;To collect your data, you’ll need to go your Facebook, &lt;code class=&quot;highlighter-rouge&quot;&gt;Settings&lt;/code&gt; then click for on &lt;code class=&quot;highlighter-rouge&quot;&gt;Download a copy&lt;/code&gt;, and download it when it is ready.&lt;br /&gt;
Then extract your &lt;code class=&quot;highlighter-rouge&quot;&gt;messages.htm&lt;/code&gt; for the archive and &lt;strong&gt;parse it&lt;/strong&gt; in text format with &lt;a href=&quot;https://github.com/ownaginatious/fbchat-archive-parser&quot;&gt;fbchat-archive-parser&lt;/a&gt;. Here is the command:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;fbcap messages.htm &lt;span class=&quot;nt&quot;&gt;-r&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;-f&lt;/span&gt; json  &lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; file.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h4 id=&quot;connecting-to-internet-to-retrieve-user-name&quot;&gt;Connecting to Internet to retrieve user name&lt;/h4&gt;
&lt;p&gt;The &lt;code class=&quot;highlighter-rouge&quot;&gt;-r&lt;/code&gt; is not obligatory, but then some conversation will be not be affiliated together. Conversations are paired given the name of the writer, and sometimes the same writer will have its name or its ID, which results in different person for the parser. With &lt;code class=&quot;highlighter-rouge&quot;&gt;-r&lt;/code&gt; IDs are replaced, as possible, with sender name.&lt;/p&gt;

&lt;p&gt;To parse your conversation nicely, you’ll need &lt;a href=&quot;https://github.com/louishenrifranc/facebook_bot/blob/master/Data/parse_messenger_chat.py&quot;&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;parse_messenger_chat.py&lt;/code&gt;&lt;/a&gt;. In the file replace &lt;code class=&quot;highlighter-rouge&quot;&gt;main_user&lt;/code&gt; with your name, and you can restrict your bot to be trained only on your answer &lt;code class=&quot;highlighter-rouge&quot;&gt;restrict_answer_to_main_user=True&lt;/code&gt; or on all answers of you and your friends.&lt;/p&gt;

&lt;p&gt;Finally, run &lt;code class=&quot;highlighter-rouge&quot;&gt;python parse_messenger_chat.py&lt;/code&gt;&lt;/p&gt;

&lt;h2 id=&quot;2-train-a-model&quot;&gt;2. Train a model&lt;/h2&gt;
&lt;p&gt;For training a model, you will use the script &lt;a href=&quot;https://github.com/louishenrifranc/facebook_bot/blob/master/main.py&quot;&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;main.py&lt;/code&gt;&lt;/a&gt;.&lt;br /&gt;
Inside the code, you can change some parameters of the model, using &lt;em&gt;Tensorflow&lt;/em&gt;  Flags:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;Number of epochs: &lt;code class=&quot;highlighter-rouge&quot;&gt;nb_epochs&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;Number of layers: &lt;code class=&quot;highlighter-rouge&quot;&gt;num_layers&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;Number of hidden neurons in cells: &lt;code class=&quot;highlighter-rouge&quot;&gt;num_hidden&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;Size of the embeddings for characters: &lt;code class=&quot;highlighter-rouge&quot;&gt;embedding_size&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;Using validation set: &lt;code class=&quot;highlighter-rouge&quot;&gt;use_attention&lt;/code&gt; (Notes: It does not make a lot of sense on character level models)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Start the training with &lt;code class=&quot;highlighter-rouge&quot;&gt;python main.py&lt;/code&gt;. 
&lt;img src=&quot;/img/2017-04-27-facebook_bot/waiting_train.jpg&quot; alt=&quot;alt text&quot; /&gt;
Go jogging, have a good sleep and come back the next days.&lt;/p&gt;

&lt;h3 id=&quot;computation-cost&quot;&gt;Computation cost&lt;/h3&gt;
&lt;p&gt;I suggested you have a good GPU and a lot of RAM on your computer. Otherwise, the model may not be able to be trained.&lt;/p&gt;

&lt;h2 id=&quot;3-run-a-bot-online&quot;&gt;3. Run a bot online&lt;/h2&gt;
&lt;p&gt;There are two solutions to branch your &lt;em&gt;“Artificial General Intelligence”&lt;/em&gt; to the Internet. Whether through a Page &lt;a href=&quot;http://masnun.com/2016/05/22/building-a-facebook-messenger-bot-with-python.html&quot;&gt;(see this tutorial)&lt;/a&gt; or through your profile.&lt;/p&gt;

&lt;h3 id=&quot;connect-through-your-profile&quot;&gt;Connect through your Profile&lt;/h3&gt;
&lt;p&gt;After training, your model should be saved in a new folder, which name is uniquely defining the architecture you used.&lt;br /&gt;
You need to find the latest trained model and locate the last Meta graph. Then, change in &lt;a href=&quot;https://github.com/louishenrifranc/facebook_bot/blob/master/bot.py&quot;&gt;&lt;code class=&quot;highlighter-rouge&quot;&gt;bot.py&lt;/code&gt;&lt;/a&gt; the &lt;code class=&quot;highlighter-rouge&quot;&gt;logdir&lt;/code&gt; and the &lt;code class=&quot;highlighter-rouge&quot;&gt;model_name&lt;/code&gt; variables.
For connecting to Messenger, I used &lt;a href=&quot;https://github.com/carpedm20/fbchat/&quot;&gt;fbchat&lt;/a&gt;. 
Finally, run the script &lt;code class=&quot;highlighter-rouge&quot;&gt;python bot.py&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Your bot is now answering all messages send to you. Messages are answered sequentially in the order of receiving. There is always a short delay :).&lt;/p&gt;

&lt;p&gt;Have fun.&lt;/p&gt;
&lt;h2 id=&quot;acknowledgments&quot;&gt;Acknowledgments&lt;/h2&gt;
&lt;p&gt;This project was done with&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/XefPatterson&quot;&gt;Felix Harvey&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/julienroy13&quot;&gt;Julien Roy&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</description>
        <pubDate>Wed, 26 Apr 2017 21:39:00 -0400</pubDate>
        <link>http://louishenrifranc.github.io/techblog/2017/04/27/facebook_bot/</link>
        <guid isPermaLink="true">http://louishenrifranc.github.io/techblog/2017/04/27/facebook_bot/</guid>
        
        <category>neural network</category>
        
        <category>python</category>
        
        <category>seq2seq</category>
        
        <category>NLP</category>
        
        <category>Facebook</category>
        
        
        <category>techblog</category>
        
      </item>
    
      <item>
        <title>Support Vector Machine, a Gentle Mathematical Introduction</title>
        <description>&lt;p&gt;As I was talking with a fellow friend, I had this conversation where he explains how Support Vector Machine had performed really well on its last biological data set compared to a neural network approach. His data set was obviously not large enough to fit the framework of crunchy data algorithm such as NN, so he set with SVM.&lt;/p&gt;

&lt;p&gt;Apart from using them with &lt;code class=&quot;highlighter-rouge&quot;&gt;scikit&lt;/code&gt;, I never had the opportunity to learn how they behave. By spending the weekend reading on SVM, here is my attempt to try filling the hole.&lt;/p&gt;

&lt;h1 id=&quot;margin-geometric-and-functionnal&quot;&gt;Margin: geometric and functionnal&lt;/h1&gt;
&lt;h2 id=&quot;introduction-to-the-problem&quot;&gt;Introduction to the problem&lt;/h2&gt;
&lt;p&gt;Given \( x \) our attribute vector, \( w \), and \( b \) the learning weights of our algorithm, we want to predict a label \(y \in \{-1, 1\}\). A simple classifier formula is defined as \(g( w^Tx + b ) \) such:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;g(z) = 1\text{ if } z &gt; 0&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;% &lt;![CDATA[
g(z) = -1\text{ if } z &lt; 0 %]]&gt;&lt;/script&gt;

&lt;h2 id=&quot;functionnal-margin&quot;&gt;Functionnal Margin&lt;/h2&gt;
&lt;p&gt;The functionnal margin is defined as \(\hat{\gamma^{i}} = y^{i}(w^Tx + b)\), where \(y^{i} \in \{-1, 1\}\). We want the margin to be as large as possible for every examples in our training set \((x_i, y_i)\), because it’s easy to see that if \(\hat{\gamma^i}\) is positive, then both \(h^i\) and \(g(z)\) are of the same sign, and if furthermore the \(\hat{\gamma}\) value is large, then the algorithm is confident about its prediciton.&lt;/p&gt;

&lt;h4 id=&quot;notes&quot;&gt;Notes&lt;/h4&gt;
&lt;p&gt;The functional margin gives the same prediction independently of the scaling of \(w\) and \(b\). It might not be fascinating but it will help us define our optimization problem.&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;functionnal margin&lt;/strong&gt; is then defined as:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\displaystyle{\min \hat{\gamma^i}}&lt;/script&gt;

&lt;p&gt;for all \(i\) pairs of training labeled data.&lt;/p&gt;

&lt;h2 id=&quot;geometric-margin&quot;&gt;Geometric Margin&lt;/h2&gt;
&lt;p&gt;Now consider a linearly separable dataset, and an linear boundary (\(H\)) between them. \(A\) is a point labelled as \(+1\).&lt;/p&gt;

&lt;h4 id=&quot;notes-1&quot;&gt;Notes&lt;/h4&gt;
&lt;p&gt;Let’s first start from our functional margin definition and prove the orthonality between \(w^T\) and (\(H\)).&lt;br /&gt;
Given two points \(x_1, x_2\) both &lt;strong&gt;on&lt;/strong&gt; the separation line, this point would have a value of \((w^Tx_1 + b) = 0\) and \((w^Tx_2 + b) = 0\) respectively.&lt;/p&gt;

&lt;p&gt;If we take the difference between this two equations we come to the conclusion:&lt;/p&gt;

&lt;p&gt;&lt;script type=&quot;math/tex&quot;&gt;w^T \cdot (x_1 - x_2) = 0&lt;/script&gt; (1)&lt;/p&gt;

&lt;p&gt;Hence \(w\) is orthogonal to any segment of the separation line.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Now, let’s try to define what the geometric margin \(\gamma^i\) is. It is the &lt;strong&gt;smallest distance&lt;/strong&gt; between the linear boundary and every point in our dataset.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Great, but how can we defined it geometrically?&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;Let’s start by computing the orthogonal projection of any point in our dataset \(x_i\) on (\(H\)):&lt;/p&gt;

    &lt;script type=&quot;math/tex; mode=display&quot;&gt;x_i - \gamma^i \frac{\vec w}{\lVert w \rVert}&lt;/script&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;We can use the orthogonality equation and inject this new point in the equation:&lt;/p&gt;

    &lt;script type=&quot;math/tex; mode=display&quot;&gt;w^T \cdot (x_i - \gamma^i \frac{\vec w}{\lVert w \rVert}) + b = 0&lt;/script&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Solving for \(\gamma^i\) gives us:&lt;/p&gt;

    &lt;script type=&quot;math/tex; mode=display&quot;&gt;\gamma^i \frac{\vec w}{\lVert w \rVert}^2 =   \vec w^T \cdot x_i + b&lt;/script&gt;

    &lt;script type=&quot;math/tex; mode=display&quot;&gt;\gamma^i =   \frac{\vec w}{(\lVert w \rVert)^2}^T \cdot x_i + \frac{b}{(\lVert w \rVert)^2}&lt;/script&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Notes that this defintion holds if our training examples have all been correctly separated. In a more general way, we define for any pair \((x_i, y_i)\), the margin as:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\displaystyle{\gamma^i =   y_i (\frac{\vec w^T}{(\lVert w \rVert)^2} \cdot x_i + \frac{b}{(\lVert w \rVert)^2})}&lt;/script&gt;

&lt;p&gt;The &lt;strong&gt;geometric margin&lt;/strong&gt; is then defined as:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\displaystyle{\min \gamma^i}&lt;/script&gt;

&lt;p&gt;for all \(i\) pairs of training labeled data.&lt;/p&gt;

&lt;h2 id=&quot;find-the-optimal-classificator&quot;&gt;Find the optimal classificator&lt;/h2&gt;
&lt;h3 id=&quot;first-formulation&quot;&gt;First formulation&lt;/h3&gt;
&lt;p&gt;Nowadays, most machine learning models are defined as an optimization problem, and Support Vector Machine is one of them.&lt;/p&gt;

&lt;p&gt;The simplest idea to formulate the problem is simply to trying to maximize the geometric margin, which in mathematical symbols is written as:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\!\max \gamma&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\text{s.c }y^i(w^T x_i + b) \gt \gamma&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\text{and }\lVert w \rVert = 1&lt;/script&gt;

&lt;p&gt;Cool, we’re done. Our problem is defined, let’s throw it to any optimizer program…. If only it were that simple because let’s be honest, this formulation is &lt;strong&gt;terrible&lt;/strong&gt;; the set of points that satisfy the norm constraint is not convex (the norm function itself is convex), hence affecting the convexity of the problem.&lt;/p&gt;

&lt;p&gt;Our problem becomes not convex, and it’s bad, because we can’t use convex optimization. Convex optimization is very robust, and on a side note, I think in deep learning there should be more research towards deep convex neural networks &lt;sup id=&quot;fnref:convex-opti&quot;&gt;&lt;a href=&quot;#fn:convex-opti&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;

&lt;p&gt;Hold on, if you don’t trust me, at least trust the Maths:&lt;/p&gt;

&lt;p&gt;Both \(\lVert w \rVert = 1\), and  \(\lVert -w \rVert = 1\) satisfy the norm constraint, however if we pick a convex sum of this two points (i.e \(\frac{1}{2} (\lVert w \rVert + \lVert -w \rVert) =  0\)) which is not on the unit circle.&lt;/p&gt;

&lt;h3 id=&quot;second-formulation&quot;&gt;Second formulation&lt;/h3&gt;
&lt;p&gt;The second formulation is simply a rewriting with the functionnal margin (i.e  \(\gamma = \frac{\hat{\gamma}}{\lVert w \rVert)}\)).&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;max \frac{\hat{\gamma}}{\lVert w \rVert}&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\text{s.c }y^i(w^T x_i + b) \gt \gamma&lt;/script&gt;

&lt;p&gt;The problem is convex in its constraint, but now we have a non-convex objective function. Let’s dive deeper.&lt;/p&gt;

&lt;h3 id=&quot;third-formulation&quot;&gt;Third formulation&lt;/h3&gt;
&lt;p&gt;Do you remember, when I said that the formulation of the functional margin was invariant w.r.t rescaling…&lt;/p&gt;

&lt;p&gt;&lt;em&gt;“Don’t speak all together”.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If \(w, b\) could be rescaled, then we can set \(\hat{\gamma} = 1\). Hence&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\frac{\hat{\gamma}}{\lVert w \rVert} = \frac{1}{\lVert -w \rVert}&lt;/script&gt;

&lt;p&gt;In our problem we are looking for the \(\displaystyle{max\frac{1}{w}}\) such that \(\forall\text{ }w^{'} \):&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\frac{1}{\lVert w \rVert} \gt \frac{1}{\lVert w^{'} \rVert}&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\lVert w^{'} \rVert \gt \lVert w \rVert&lt;/script&gt;

&lt;p&gt;We square the norm, without changing the correctness of the inequation. Hence we are turning our initial nonconvex maximization problem, into a quadratic minimization problem.&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\lVert w^{'} \rVert^2 \gt \lVert w \rVert^2&lt;/script&gt;

&lt;p&gt;To keep the derivative nice, we can add \(\frac{1}{2}\)&lt;/p&gt;

&lt;p&gt;And here is our final optimization formulation :&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;min  \frac{1}{2}{\lVert w \rVert}^2&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\text{s.c }y^i(w^T x_i + b) \gt 1&lt;/script&gt;

&lt;p&gt;This problem is now convex in its constraint, and the objective function is also convex. Even if we are happy with this solution, we can still be smarter and avoid some computations, while extending this formulation to nonseparable hyperplane. Here is all the magic of SVM.&lt;/p&gt;

&lt;h2 id=&quot;passing-into-dual-form&quot;&gt;Passing into dual form&lt;/h2&gt;
&lt;p&gt;We will use Lagrange polynomial. If you don’t know what Lagrange problem and dual form are, I’ll make an article about it soon. Let’s first transform our inequality constraint into a function \(g_i(w) : y^i(w^T x_i + b) - 1 = 0\) for all \(i\) pair of examples.&lt;/p&gt;

&lt;p&gt;We can then write the loss function as \( \displaystyle{L(w, b, \alpha) = \frac{1}{2}{\lVert w \rVert}^2 + \sum_{i=1}^n \alpha_i y^i(w^T x_i + b) - 1  }\) where \(\alpha_i\) are the Lagrange multipliers.&lt;/p&gt;

&lt;p&gt;If we try to find an extrema point, we will have to compute the partial derivative concerning the weight and the bias, and set it equal to zero.&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\frac{\partial L}{\partial w} = w - \sum_{i=1}^N \alpha_i y^i(w^T x_i + b) - 1 = 0&lt;/script&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\frac{\partial L}{\partial b} = - \sum_{i=1}^N \alpha_i y^i = 0&lt;/script&gt;

&lt;p&gt;Plugging this back into the \(L\), we find at:&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;\displaystyle{L(w, b, \alpha) = \frac{1}{2}\sum_{i,j=1}^n \alpha_i \alpha_j y^i y^j x_i^ T x_j + \sum_{i=1}^N \alpha_i}&lt;/script&gt;

&lt;p&gt;This will be the function will be maximizing for our dual formulation under the constraint of \(\sum_{i=1}^N \alpha_i y^i = 0\).&lt;/p&gt;

&lt;h4 id=&quot;sides-notes-1&quot;&gt;Sides notes 1&lt;/h4&gt;
&lt;p&gt;By definition &lt;strong&gt;support vector&lt;/strong&gt; are the closest points to the linear boundary. For those points \(x_i\), the Lagrange multiplier \(\alpha_i\) can be anything because the functional margin is equal to 1, hence \(g_i(w) = 0\) .&lt;/p&gt;

&lt;h2 id=&quot;kernel&quot;&gt;Kernel&lt;/h2&gt;
&lt;h3 id=&quot;definition&quot;&gt;Definition&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;Kernel are magic. Kernel are life. Kernel trick.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Features are computed by transforming our initial attribute vector \(w\) into a different dimensionnal space. Feature mapping (i.e for example \(\theta(x) = x^T \cdot x\)) passes attributes in new space.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A kernel is defined as an inner product between feature mapping.&lt;/strong&gt;&lt;/p&gt;

&lt;h3 id=&quot;how-to-used-them-in-the-context-of-svm&quot;&gt;How to used them in the context of SVM&lt;/h3&gt;
&lt;p&gt;If we take the formulation of our dual problem, we see one inner product between the attributes vector. Now with our kernel definition, we can replace this inner product \(&amp;lt;x,x&amp;gt;\) by an inner product of feature mappings \(&amp;lt;\theta(x), \theta(x)&amp;gt;\). Math doesn’t change, convex properties still hold, initial data is only mapped into a new space.&lt;/p&gt;

&lt;h3 id=&quot;the-kernel-trick&quot;&gt;THE Kernel trick&lt;/h3&gt;
&lt;p&gt;Now the trick about kernel is that feature mapping can be computationally expensive to compute. For example, in linear regression, we sometimes transform our attributes by replacing \(x\) with all its monomes up to a certain degree. For three attributes \(A={a_1, a_2, a_3}\), and up to degree 2, we’ll have a vector of dimension \( |A|^2\):&lt;/p&gt;

&lt;script type=&quot;math/tex; mode=display&quot;&gt;F = \begin{bmatrix}
           a_1a_1 \\
           a_1a_2 \\
           a_1a_3 \\
           \vdots \\
           a_3a_3 \\
         \end{bmatrix}&lt;/script&gt;

&lt;p&gt;And it’s only degree 2. However, if we can find a kernel function which inner product has feature mapping equal F, then computation will be much faster \(O(|A|)\). 
With some simple math, we can find such a kernel K, \(K(x,z) = (x^Tz)^2 = \dots = \displaystyle{\sum_{i,j = 1}^{|A|}(x_ix_j)(z_iz_j)}\). And &lt;strong&gt;here is the magic&lt;/strong&gt;. We only need to compute a function which “under the hood” is an inner product of feature mapping without even having to compute this potentially large vector.&lt;/p&gt;

&lt;p&gt;Some famous kernel in the literature: the &lt;strong&gt;Gaussian kernel&lt;/strong&gt; is one of them. It has been proved to map attributes to an infinite space.&lt;/p&gt;

&lt;p&gt;Sometimes, it’s just enough to know it’s a kernel, and you don’t want to check if it is defined as an inner product. &lt;strong&gt;Mercel theorem&lt;/strong&gt; states that:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Theorem&lt;/strong&gt;: If K is a positive symmetric semi definite matrix, then K is also a valid kernel, so that there exists a feature mapping \(\theta\) such that \(K(x, z) = \theta(x)^T\theta(y)\).&lt;/p&gt;

&lt;h2 id=&quot;what-if-the-optimal-margin-was-not-the-best&quot;&gt;What if the optimal margin was not the best.&lt;/h2&gt;
&lt;p&gt;In Machine Learning, in the context of optimization problem, finding the global optimum is sometimes not necessary, and can lead to bad generalization on unseen data. Hence it is possible to add a regularization term \(\displaystyle{C\sum_{i=1}^n \epsilon_i}\) in the primal formulation where \(\epsilon_i\) replaces our \(\hat{\gamma} = 1\).&lt;/p&gt;

&lt;p&gt;It allows some i pairs to have a margin smaller than 1, but not too often, neither “too much”\(y_i(w^T + b) \gt 1 - \epsilon_i \).&lt;/p&gt;

&lt;p&gt;In the context of the dual formulation, a constraint is add to \(\alpha_i\) so that \(\alpha_i \lt C\).&lt;/p&gt;

&lt;h1 id=&quot;bibliography&quot;&gt;Bibliography&lt;/h1&gt;
&lt;div class=&quot;footnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:convex-opti&quot;&gt;
      &lt;p&gt;Convex Neural Networks from &lt;em&gt;Yoshua Bengio, Nicolas Le Roux, Pascal Vincent, Olivier Delalleau, Patrice Marcotte&lt;/em&gt;&amp;nbsp;&lt;a href=&quot;#fnref:convex-opti&quot; class=&quot;reversefootnote&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;
</description>
        <pubDate>Fri, 21 Apr 2017 21:39:00 -0400</pubDate>
        <link>http://louishenrifranc.github.io/mathblog/2017/04/22/support-vector-machine/</link>
        <guid isPermaLink="true">http://louishenrifranc.github.io/mathblog/2017/04/22/support-vector-machine/</guid>
        
        <category>math</category>
        
        <category>svm</category>
        
        <category>machine-learning</category>
        
        <category>optimization</category>
        
        
        <category>mathblog</category>
        
      </item>
    
  </channel>
</rss>
