Leveraging Structure for Efficient and Dexterous Contact-Rich Manipulation