Running MoveIt Pro as a Distributed System

By default, MoveIt Pro is configured to run on a single computer.

However, in some cases it can be useful to distribute the computational load across multiple computers (e.g., hardware drivers or perception modules).

When using multiple computers it is important to keep in mind that all of them must share a network and have compatible DDS configurations to ensure fast and robust communication via ROS.

In the event that a multiple computer setup is preferred, there are a few configuration changes that must be made.

Configuring CycloneDDS for Multiple PC setup

Note

The following section provides basic instructions on how to configure DDS settings for a two PC setup, one running the Agent and one running the Robot Drivers. For more information, please refer to this guide on Configuring DDS for MoveIt Pro.

To enable interprocess communication between machines in your ROS network, you must configure CycloneDDS. This can be done by creating a ~/.ros/cyclonedds.xml or exporting the following variables:

  • CYCLONEDDS_NETWORK_INTERFACE
  • CYCLONEDDS_PEER_ADDRESSES
  • CYCLONEDDS_USE_MULTICAST

The values to use for CYCLONEDDS_NETWORK_INTERFACE and CYCLONEDDS_PEER_ADDRESSES depend on the configuration of the network interfaces the Agent Computer and the Driver Computer use to communicate with each other.

  • For each system, CYCLONEDDS_NETWORK_INTERFACE needs to be set to the name of the network interface which is used to connect to the ROS network.
  • For each system, CYCLONEDDS_PEER_ADDRESSES needs to be set to a comma-separated list containing the IP addresses of the Agent Computer and Driver Computer.

You can find the current names of the network interfaces on each system by running ip a from the command line. The output from this command will be different for the Agent Computer and the Driver Computer.

For example, suppose our system consists of two computers:

  • An “Agent Computer” which runs MoveIt Pro.
  • A “Driver Computer” running a real-time operating system for communicating with the hardware.

The Agent Computer has two network interfaces (one to connect to the internet and another to connect to the Driver Computer), so the network interface information would look similar to this:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 1c:69:7a:0f:3e:75 brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
    inet #.#.#.#/24 brd #.#.#.255 scope global dynamic noprefixroute eno1
       valid_lft 68323sec preferred_lft 68323sec
    inet6 #::#:#:#:#/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f4:3a:8c:70:c2:b1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.10/24 brd 192.168.10.255 scope global noprefixroute enp0s31f6
      valid_lft forever preferred_lft forever
    inet6 fe80::c8e:a7ff:f36f:1ef4/64 scope link noprefixroute
      valid_lft forever preferred_lft forever

The Driver Computer uses two wired network interfaces (one to connect to the Agent Computer and another to connect to the robot’s Control Box), so its network interface information would look like this:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: enp112s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f4:3a:8c:70:c2:b1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.12/24 brd 192.168.10.255 scope global noprefixroute enp112s0
      valid_lft forever preferred_lft forever
    inet6 fe80::6a48:1686:19d8:3085/64 scope link noprefixroute
      valid_lft forever preferred_lft forever
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether f4:3a:8c:70:c2:b1 brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
    inet 192.10.0.12/24 brd 192.10.0.255 scope global noprefixroute eno1
      valid_lft forever preferred_lft forever
    inet6 fe80::a6ae:11ff:fe1e:cb89/64 scope link noprefixroute
      valid_lft forever preferred_lft forever

Using the example output above, we would export the following entries to the Agent Computer environment:

CYCLONEDDS_NETWORK_INTERFACE=enp0s31f6
CYCLONEDDS_PEER_ADDRESSES=192.168.10.10,192.168.10.12

And export the following entries to the Driver Computer:

CYCLONEDDS_NETWORK_INTERFACE=enp112s0
CYCLONEDDS_PEER_ADDRESSES=192.168.10.10,192.168.10.12

For persistent changes, create and edit a ~/.ros/cyclonedds.xml and add export CYCLONEDDS_URI=~/.ros/cyclonedds.xml to your ~/.bashrc. Run moveit_pro configure and select Y when asked Do you want to replicate the localhost DDS configuration?

An example configuration would look like this:

<?xml version="1.0" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
  <Domain id="any">
    <General>
      <Interfaces>
        <NetworkInterface name="lo"/>
      </Interfaces>
      <AllowMulticast>false</AllowMulticast>
    </General>
    <Discovery>
      <ParticipantIndex>auto</ParticipantIndex>
      <Peers>
        <Peer Address="127.0.0.1"/>
      </Peers>
      <MaxAutoParticipantIndex>1000</MaxAutoParticipantIndex>
    </Discovery>
  </Domain>
</CycloneDDS>

Configuring Cameras for Multiple PC setup

Suppose you are running the MoveIt Pro Agent on a machine (the Agent machine) and the robot drivers on another (the Driver machine). Normally, the cameras should be started in the top level launch file agent_bridge.launch.xml in the configuration package. Therefore, the cameras should be connected to the Agent machine. If the cameras need to be connected to the Drivers machine (for example, due to wiring constraints), then add the camera launch file to the drivers launch file.