Cell shaping is to configure the radio antenna parameters (which in our case, electrical downtilt) to improve the service quality. In this talk, we introduce a Reinforcement Learning (RL) agent that automates the cell shaping. More precisely, the agent aims at tuning tilt of multiple antennas to be generalized and adaptive to the random traffic demand/hotspots in urban map. The training has been accelerated by leveraging RLlib's scalability which is an open source distributed RL library driven by UC Berkeley RISELab. RLlib's support for distributed execution is critical since our radio simulator is very compute intensive. We show that the RL agent’s tuning is well adapted to any scenario, and thereby, outperforms the baseline algorithms.