Reinforcement learning has shown promise performance in automatic trading and financial Portfolio Management(PM). However, existing methods cannot effectively extract the common trend from the historical data of an asset, nor can they leverage the unique properties of different assets. In this paper, we propose Attention ActorRNN, an attention-based deep reinforcement learning framework for smart portfolio management. A temporal attention mechanism is presented to identify the fluctuation pattern on different trading days, and a stock attention mechanism is presented to identify the influence of different assets on the portfolio. Both the attention mechanisms are jointly incorporated into the Actor only model which is realized by a Recurrent Neural Network(RNN) and obtain improved performance in bear, bull, or shocking market of Chinese A-share market, respectively. Our Attention ActorRNN achieves state-of-the-art performance in the Chinese A-share market, achieving a Sharpe ratio of 2.31, final accumulated portfolio value of 126.47 above the baseline. In addition, we propose a Multi-window ActorCNN to take advantages of Convolutional Neural Network (CNN) for the Actor model in local feature extraction and computational parallelism. It complements the Attention ActorRNN as an alternative model for investors in specific market states of the A-share market.