Number Theory In 3D Gravity And From 4D Gauge Theory