{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Robotic arm control with MuJoCo\n", "\n", "In this tutorial we will demonstrate how to control a robotic arm using the Collimator library and the MuJoCo physics engine. We will use the Franka Emika Panda robotic arm to pick up a block and set it on a table. The multibody dynamics are handled with MuJoCo, but the controller is implemented in Collimator. This can be done either in the UI or directly with the Python library. This notebook is a standalone implementation with the Python library, but at the end we will also note how to do the same thing in the UI." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from scipy.spatial import transform\n", "import matplotlib.pyplot as plt\n", "\n", "import jax\n", "import jax.numpy as jnp\n", "\n", "import collimator\n", "from collimator import library\n", "from collimator.backend import io_callback\n", "\n", "# If needed, enable offscreen rendering (eg. for CI)\n", "# import os\n", "# os.environ['MUJOCO_GL'] = 'egl'\n", "\n", "import mujoco\n", "import mediapy" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get started, we will need to download and extract the [MuJoCo scene definition](assets/franka_emika_panda.zip): a collection of XML and STL files that define the robot and surrounding environment. This .zip file should be extracted to a directory called `assets` in the same directory as the notebook." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to render the scene and videos, you will also need to have ffmpeg installed. If not found automatically, the path can be set as follows:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# Uncomment and change to your ffmpeg path, if needed:\n", "# mediapy.set_ffmpeg(\"/opt/homebrew/bin/ffmpeg\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's start by checking that we can directly load and render the scene. We should see the robotic arm in its \"home\" position about halfway between the orange \"target\" block and the grey \"table\" block. Our goal will be to pick up the block and set it on the table." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "collimator:INFO Initialized MuJoCo from model file 'assets/franka_emika_panda/scene.xml' with dt=0.01, 16 generalized positions, 15 generalized velocities, and 0 actuators. The default control input vector (nu=8) is:\n", "np.array([0.0, 0.0, 0.0, -1.57079, 0.0, 1.57079, -0.7853, 255.0])\n", "libdecor-gtk-WARNING: Failed to initialize GTK\n", "Failed to load plugin 'libdecor-gtk.so': failed to init\n" ] }, { "data": { "text/html": [ "