Robo Quest

Issac Sim provides a set of APIs that are relevant for robotics applications. They abstract the complexities of dealing with Pixar's USD Python API.

Core API

BaseSample – A boilerplate extension application that sets up the basics for a robotic extension application.

An extension application is a UI elements provider in Issac Sim parlance.

BaseSample performs

• Loading the world and it's assets.
• Clearing the world when a new stage is added.
• Reset the world to default state.
• Handle hot reloading.

BaseSample has a World instance. World allows to interact with the simulator in an easy and modular way. It handles time related events(including physics step), resetting scene and adding tasks etc. World is a singleton and the reference to it can be obtained anywhere as World.instance() (after the BaseSample has been initialized in the constructor).

A World contains an instance of a Scene. Scene manages simulation assets in the USD Stage. Scene provides an API to manipulate different USD assets in the stage.

When creating a BaseSample one can use setup_scene to initialize scene for the first time use.

NOTE: setup_scene is not called during hot reloading but only during loading the world from an EMPTY stage.

Objects can be added to the scene using scene.add()

setup_post_load is called when load button is pressed, after setup_scene and after one physics time step.

When pressing the play button for simulation we can get a registered callback triggered using World's add_physics_callback which takes the callback function as the parameter. This function is called just before the physics time step is executed.

Standalone application

Instead of an extension application one can also create a standalone application where the application is stated from python and one can control when to step physics and rendering.

After creating a SimulationApp one can add the world, scene and objects in the scene as usual. Then using world.step(render=True) one can execute a single physics time step.

Adding a robot to the stage

Issas Sim provides a number of helper classes which make it easier to deal with USD assets. After adding the robot as USD to the stage using add_reference_to_stage one can wrap the USD asset under Robot and add Robot to the scene.

Robot is an Articulation which according to the comments in the source code provides high level functions to deal with an articulation prim and its attributes/ properties.”

Every Articulation has an ArticulationController which according to it's comments is a PD Controller of all degrees of freedom of an articulation, can apply position targets, velocity targets and efforts.

During a physics step(which we have access to using callbacks as state above) we can apply an ArticulationAction to the ArticulationController to specify joint positions, efforts or velocities.

Instead of using the Robot class one can also use a more specialized version called WheeledRobot which derives from the Robot class. This class then provides a more specialized apply_wheel_action instead of more general apply_action of the Robot class.

Adding a custom controller

Controllers inherit from BaseController and must implement forward method which receive a whatever command and returns ArticulationAction.

Once a custom controller has been created, this can be assigned as the controller of the robot. The robot can apply_action on the ArticulationAction as returned by the forward method.

Before implementing custom controllers it is always wise to check ready made controllers such as DifferentialController or WheelBasePoseController.

Manipulator robots have their own domain specific controllers such as PickAndPlaceController. The forward method of this controller for example takes a picking position, a placing position and the current joint position and calculates a step forward and returns an ArticulationAction which the robot can apply.

Tasks

For creating complex scenes one can use Tasks which provides a way to modularize scene creation. Custom tasks derive from BaseTask and implement get_observationswhich Returns current observations from the objects needed for the behavioral layer. The task can implement setup_scene pre_step and post_reset etc as usual.

Task can be added to the World which will then call task's setup_scene when initializing the scene.

Tasks help encapsulate the logic and hence make it easier to write larger more complex scenes. Just as in adding custom controllers, Issac Sim provides common tasks which should be used if relevant. For example PickPlace task for Franka.

Tasks can have other tasks in them. In this case the subtask's setup_scene can be called by the parent task.

Personally, I feel Task is not aptly named. It should be called SubScene perhaps.

Multi robot scene

Just a matter of maintaining a state machine and orchestrate the tasks as needed in the physics time step. Each robot has it's own controller and requires an ArticulationAction as needed to make the whole scene and handoff work as needed.