Intra-temporal attention on input sequence while decoding