Jake Varley
Jake Varley
Software Engineer · Robotics, Google DeepMind NYC

I am a Software Engineer on the Google DeepMind Robotics team led by Carolina Parada, collaborating closely with Vikas Sindhwani in the NYC lab. My current focus is on safety and alignment for humanoid robots—developing the models, frameworks, and evaluation methods needed to deploy capable humanoid systems safely in real-world environments alongside humans. This includes work on semantic safety, uncertainty-aware planning, and building robustly safe controllers that interface with modern foundation models.

I received my Ph.D. from Columbia University in 2018, where I was advised by Peter Allen. My broader research interests span high-DoF planning and control, 3D computer vision, reinforcement learning, and foundation models for robotics. I enjoy tackling the engineering challenges that come with the multidisciplinary nature and system complexity inherent to real-robot research.

Publications
2025
Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer
Gemini Robotics Team (incl. Jake Varley)
arXiv, 2025
Gemini Robotics: Bringing AI into the Physical World
Gemini Robotics Team (incl. Jake Varley)
arXiv, 2025
Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Connor Schenck et al. (incl. Jake Varley)
arXiv, 2025
2024
Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity
Jake Varley, Sumeet Singh, Deepali Jain, Krzysztof Choromanski, Andy Zeng, Somnath Basu Roy Chowdhury, Avinava Dubey, Vikas Sindhwani
IROS 2024
Modeling the Real World with High-Density Visual Particle Dynamics
William F. Whitney, Jake Varley, Deepali Jain, Krzysztof Choromanski, Sumeet Singh, Vikas Sindhwani
CoRL 2024
SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention
Isabel Leal, Krzysztof Choromanski, Deepali Jain, Avinava Dubey, Jake Varley, Michael Ryoo, Yao Lu, Frederick Liu, Vikas Sindhwani, Quan Vuong, Tamas Sarlos, Ken Oslund, Karol Hausman, Kanishka Rao
ICRA 2024
How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies
Montserrat Gonzalez Arenas, Ted Xiao, Sumeet Singh, Vidhi Jain, Allen Z. Ren, Quan Vuong, Jake Varley, Alexander Herzog, Isabel Leal, Sean Kirmani, Dorsa Sadigh, Vikas Sindhwani, Kanishka Rao, Jacky Liang, Andy Zeng
ICRA 2024
2023
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners Best Student Paper · CoRL 2023
Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, Anirudha Majumdar
CoRL 2023
Visual Backtracking Teleoperation: A Data Collection Protocol for Offline Image-Based Reinforcement Learning
David Brandfonbrener, Stephen Tu, Avi Singh, Stefan Welker, Chad Boodoo, Nikolai Matni, Jake Varley
ICRA 2023
2022
Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation
Xuesu Xiao, Tingnan Zhang, Krzysztof Choromanski, Tsang-Wei Edward Lee, Anthony Francis, Jake Varley, Stephen Tu, Sumeet Singh, Peng Xu, Fei Xia, Sven Persson, Dmitry Kalashnikov, Leila Takayama, Roy Frostig, Jie Tan, Carolina Parada, Vikas Sindhwani
CoRL 2022
Implicit Kinematic Policies: Unifying Joint and Cartesian Action Spaces in End-to-End Robot Learning
Aditya Ganapathi, Pete Florence, Jake Varley, Kaylee Burns, Ken Goldberg, Andy Zeng
ICRA 2022
Multiscale Sensor Fusion and Continuous Control with Neural CDEs
Sumeet Singh, Francis McCann Ramirez, Jacob Varley, Andy Zeng, Vikas Sindhwani
IROS 2022
Mobile Manipulation Leveraging Multiple Views
David Watkins-Valls, Peter K. Allen, Henrique Maia, Madhavan Seshadri, Jonathan Sanabria, Nicholas R. Waytowich, Jacob Varley
IROS 2022
2021
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Ben Eysenbach, Ryan Julian, Chelsea Finn, Sergey Levine
ICML 2021
Visionary: Vision Architecture Discovery for Robot Learning
Iretiayo Akinola, Anelia Angelova, Yao Lu, Yevgen Chebotar, Dmitry Kalashnikov, Jacob Varley, Julian Ibarz, Michael Ryoo
ICRA 2021
Reward Machines for Vision-Based Robotic Manipulation
Alberto Camacho, Jacob Varley, Andy Zeng, Deepali Jain, Atil Iscen, Dmitry Kalashnikov
ICRA 2021
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, Karol Hausman
arXiv, 2021 — also accepted to CoRL 2021 as "Scaling Up Multi-Task Robotic Reinforcement Learning"
2020
Ode to an ODE
Krzysztof Choromanski, Jared Quincy Davis, Valerii Likhosherstov, Xingyou Song, Jean-Jacques Slotine, Jacob Varley, Honglak Lee, Adrian Weller, Vikas Sindhwani
NeurIPS 2020
Generative Attention Learning: a "GenerAL" Framework for High-Performance Multi-Fingered Grasping in Clutter
Bohan Wu, Iretiayo Akinola, Abhi Gupta, Feng Xu, Jacob Varley, David Watkins-Valls, Peter K. Allen
Autonomous Robots, vol. 44, 2020
Learning Precise 3D Manipulation from Multiple Uncalibrated Cameras
Iretiayo Akinola, Jacob Varley, Dmitry Kalashnikov
ICRA 2020
2019
MAT: Multi-Fingered Adaptive Tactile Grasping via Deep Reinforcement Learning
Bohan Wu, Iretiayo Akinola, Jacob Varley, Peter K. Allen
CoRL 2019
Teleoperator Imitation with Continuous-Time Safety
Bachir El Khadir, Jacob Varley, Vikas Sindhwani
RSS 2019
Multi-Modal Geometric Learning for Grasping and Manipulation
David Watkins-Valls, Jacob Varley, Peter K. Allen
ICRA 2019
2018
Workspace Aware Online Grasp Planning
Iretiayo Akinola, Jacob Varley, Boyuan Chen, Peter K. Allen
IROS 2018
Human Robot Interface for Assistive Grasping
David Watkins-Valls, Chaiwen Chou, Caroline Weinberg, Jacob Varley, Kenneth Lyons, Sanjay Joshi, Lynne Weber, Joel Stein, Peter K. Allen
arXiv, 2018
2017
Shape Completion Enabled Robotic Grasping
Jacob Varley, Chad DeChant, Adam Richardson, Joaquín Ruales, Peter K. Allen
IROS 2017
Task Level Hierarchical System for BCI-enabled Shared Autonomy
Iretiayo Akinola, Boyuan Chen, Jonathan Koss, Aalhad Patankar, Jake Varley, Peter K. Allen
IEEE-RAS Humanoid Robots 2017
2015
Generating Multi-Fingered Robotic Grasps via Deep Learning
Jacob Varley, Jonathan Weisz, Jared Weiss, Peter K. Allen
IROS 2015