Attention Mechanisms: Stop Compressing, Start Looking Back

"The art of being wise is the art of knowing what to overlook." — William James The Bottleneck We Didn't Notice In my last post , we gave networks memory. An LSTM reads a sentence word by word, maintaining a hidden state that carries context forward. It solved the forgetting problem that plagued vanilla RNNs. But there are three problems LSTM still doesn't solve. And I didn't fully understand them until I thought about my own experience learning English. I studied in Tamil medium all the way thr