As machine learning has started to shift towards deployment in the physical world, its rapid development is coupled with as much risk as benefits. In this talk, I will discuss two key aspects of developing trustworthy autonomy: generalizability and safety. I will introduce our recent progress in both mathematical foundations and real-world applications. I will also share the lessons we learned and our outlook for future research directions.