Exploiting Structures in Reinforcement Learning: Multi-Agent Homogeneity, Euclidean Symmetry, and Natural Languages