Separate communication channels for multi robot setup (one local per robot, and one global between robots)

[Tuntenfisch]

Hello,

I have a setup of multiple robots, which communicate with each other. Their internal software stack is based on ROS2 Humble. The communication between the robots is also using ROS2 Humble. The middleware used is FastDDS.

Right now, all robots see all other robots topics, i.e. they not only see topics meant for shared communication between robots, but also topics other robots use to make their internal software stack work. This kind of clutters the global communication space.

Ideally, I would want to have ROS2 topics/services that start with "/global/" be shared, and any other topic be local to the individual robot. I tried doing this with DDS router and the following config:

allowlist:
  - name: rt/global/*
  - name: rq/global/*
  - name: rr/global/*

participants:
  - name: local
    kind: local
    domain: 1
    discovery: true

  - name: global
    kind: local
    domain: 0

specs:
  discovery-trigger: any

My understanding is that this effectively bridges any topics/services starting with "/global/" between the domain ids "1" and "2". To extend this to an "arbitrary" amount of robots, I would deploy one DDS router with the above config on each robot, making sure each robot operates on a unique domain id, i.e. I would change the "local" participants domain id to something unique for that robot and run any ROS nodes using this unique id, too.

I have a few questions, tho:

1. Is the split between global and local topics as I've described advisable?
2. Does DDS router make sense for this approach?
3. Are there other, easier alternatives to achieve the same thing?
4. I don't see any easy way to automatically assign unique domain ids to robots. Of course, I can manually specify ids on each robot's system but my software stack is containerized and I would like to automate as much as possible. Any recommendations?
5. With the above config, I cannot use remove-unused-entities: true because it is incompatible with discovery-trigger: any. Ideally I would like to have unused entities removed but then the DDS router doesn't bridge any topic unless someone on the domain where the topic is published also subscribes to it first. I suppose there is no workaround for this?

[juanlofer-eprosima]

Hi @Tuntenfisch,

1. Sure it could make sense in some scenarios.
2. It would be an option to achieve your goal I believe.
3. Probably, can't think of any at the moment. I suggest you use a default XML profile configuration in your robots using ignore participant flags, so that ROS 2 nodes only communicate in the same host. Then you could have a DDS-Router in each robot, with a local participant in a local domain (could be the same in all robots since communication is only local), adding also the corresponding ignore participant flag, and another participant in the global shared domain. By the way, you could use an allowlist rule like "*/global/*" instead.
4. See previous answer.
5. No there is no workaround. Why would you like to forward data if there is no subscriber to receive it?

[Tuntenfisch]

Hi @Tuntenfisch,

1. Sure it could make sense in some scenarios.
2. It would be an option to achieve your goal I believe.
3. Probably, can't think of any at the moment. I suggest you use a default XML profile configuration in your robots using ignore participant flags, so that ROS 2 nodes only communicate in the same host. Then you could have a DDS-Router in each robot, with a local participant in a local domain (could be the same in all robots since communication is only local), adding also the corresponding ignore participant flag, and another participant in the global shared domain. By the way, you could use an allowlist rule like "*/global/*" instead.
4. See previous answer.
5. No there is no workaround. Why would you like to forward data if there is no subscriber to receive it?

Thank you for your feedback, regarding your points:

3. I think you are saying I could use the same ROS_DOMAIN_ID for each robot if I restrict the communication to local host only. This would effectively prevent me from getting any topic information (for example) from outside the robot because its communication never leaves the local host. But I suppose in certain scenarios it would be beneficial to have the ability to do something like ROS_DOMAIN_ID=<specific robot domain ID> ros2 topic list from a different machine to investigate a specific robots communication for debugging. Is my understanding of this limitation correct?

‎

5. During my testing I noticed that if I set discovery-trigger: reader and I attempt to do ros2 topic echo <topic published in another ROS_DOMAIN_ID but routed via the DDS router> the topic cannot be found unless I first create a subscriber that subscribes to this topic in the original ROS_DOMAIN_ID where that topic is published. I suppose this is because the DDS router doesn't create an internal reader/writer for routing the topic unless a reader is specifically created in the publishing ROS_DOMAIN_ID for a given topic. But if I solely want to publish a topic into the "shared" ROS_DOMAIN_ID without being interested in the published information in my own per-robot domain, I cannot do that. Or am I missing something?

A slightly off topic question:

I'm using ROS2 Humble and was wondering which DDS router version i should be using. I think ROS2 Humble is using DDS v2.6.2 so I should probably choose the DDS router version compatible with this DDS version?

Also, the installation guide I found here mentions I can install using one of the following methods:

1. Colcon: I guess this would require me to then subsequently source the workspace before being able to use the router. It's also mentioned that if a DDS implementation is already installed, not all packages need to be built. Since ROS2 Humble is using Fast DDS I guess I wouldn't need to build Fast DDS again. But where does ROS2 Humble store it's DDS installation? Is source /opt/ros/humble/setup.bash sufficient?
2. CMake (local): I could potentially also skip building certain parts when I already have them installed via ROS2 Humble, correct? Is the procedure the same?
3. CMake (system-wide): Same question as for the local installation.

I suppose there is no right or wrong installation procedure to use. I'm setting up my software stack using a docker container so I would probably go with the system-wide installation because it doesn't require sourcing the workspace like the other two do.

This is quite a long post and I very much appreciate your help.

[Tuntenfisch]

Alright, I went down the route of integrating a dedicated docker container alongside my docker container for the robot software stack using docker compose. I went with a custom dockerfile instead of the docker image provided in the documentation because the one provided isn't available through any repository:

fast_dds_router.dockerfile

FROM ubuntu:22.04

# Use bash as shell.
SHELL ["/bin/bash", "-c"]

ARG ROS_DOMAIN_ID
ENV ROS_DOMAIN_ID=${ROS_DOMAIN_ID}

USER root

#########################################################################################################
# Update and install required apt and pip dependencies.                                                 #
#########################################################################################################
RUN --mount=type=cache,id=apt_cache_fast_dds_router,target="/var/cache/apt" \
    apt update && apt install -y --no-install-recommends \
        make \
        cmake \
        g++ \
        pip \
        wget \
        git \
        gettext \
        libasio-dev \
        libtinyxml2-dev \
        libssl-dev \
        libyaml-cpp-dev && \
    apt -y autoremove && apt clean autoclean && rm -rf "/var/lib/apt/lists/*"

RUN pip install \
        vcstool

#########################################################################################################
# Install Fast DDS Router and its Dependencies.                                                         #
#########################################################################################################
ARG DDS_ROUTER_VERSION=v3.1.0
ARG DDS_ROUTER_DIRECTORY=/opt/DDS-Router
WORKDIR "${DDS_ROUTER_DIRECTORY}"

RUN mkdir \
        "src" \
        "build" && \
    wget "https://raw.githubusercontent.com/eProsima/DDS-Router/${DDS_ROUTER_VERSION}/ddsrouter.repos" && \
    vcs import "src" < "ddsrouter.repos"

ENV CMAKE_BUILD_PARALLEL_LEVEL=$(nproc)

# Foonathan Memory Vendor
RUN mkdir "build/foonathan_memory_vendor" && \
    cd "build/foonathan_memory_vendor" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/foonathan_memory_vendor" \
        -DCMAKE_INSTALL_PREFIX="/usr/local/" \
        -DBUILD_SHARED_LIBS=ON && \
    cmake --build "." --target install -j$(nproc)

# Fast CDR
RUN mkdir "build/fastcdr" && \
    cd "build/fastcdr" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/fastcdr" && \
    cmake --build "." --target install -j$(nproc)

# Fast DDS
RUN mkdir "build/fastdds" && \
    cd "build/fastdds" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/fastdds" && \
    cmake --build "." --target install -j$(nproc)

# CMake Utils
RUN mkdir "build/cmake_utils" && \
    cd "build/cmake_utils" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/dev-utils/cmake_utils" && \
    cmake --build "." --target install -j$(nproc)

# C++ Utils
RUN mkdir "build/cpp_utils" && \
    cd "build/cpp_utils" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/dev-utils/cpp_utils" && \
    cmake --build "." --target install -j$(nproc)

# DDS Pipe Core
RUN mkdir "build/ddspipe_core" && \
    cd "build/ddspipe_core" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/ddspipe/ddspipe_core" && \
    cmake --build "." --target install -j$(nproc)

# DDS Pipe Participants
RUN mkdir "build/ddspipe_participants" && \
    cd "build/ddspipe_participants" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/ddspipe/ddspipe_participants" && \
    cmake --build "." --target install -j$(nproc)

# DDS Pipe YAML
RUN mkdir "build/ddspipe_yaml" && \
    cd "build/ddspipe_yaml" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/ddspipe/ddspipe_yaml" && \
    cmake --build "." --target install -j$(nproc)

ENV LD_LIBRARY_PATH=/usr/local/lib/:${LD_LIBRARY_PATH}

# DDS Router Core
RUN mkdir "build/ddsrouter_core" && \
    cd "build/ddsrouter_core" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/ddsrouter/ddsrouter_core" && \
    cmake --build "." --target install -j$(nproc)

# DDS Router YAML
RUN mkdir "build/ddsrouter_yaml" && \
    cd "build/ddsrouter_yaml" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/ddsrouter/ddsrouter_yaml" && \
    cmake --build "." --target install -j$(nproc)

# DDS Router Tool
RUN mkdir "build/ddsrouter_tool" && \
    cd "build/ddsrouter_tool" && \
    cmake "${DDS_ROUTER_DIRECTORY}/src/ddsrouter/tools/ddsrouter_tool" && \
    cmake --build "." --target install -j$(nproc)

#########################################################################################################
# Run DDS Router.                                                                                       #
#########################################################################################################
COPY ".devcontainer/config/dds/fast_dds_router_config.yml.template" "${DDS_ROUTER_DIRECTORY}/fast_dds_router_config.yml.template"
RUN envsubst < "fast_dds_router_config.yml.template" > "fast_dds_router_config.yml"
CMD [ "ddsrouter", "-c", "fast_dds_router_config.yml" ]

The config file I use with DDS router can be seen below:

fast_dds_router_config.yml.template

version: v5.0

allowlist:
  - name: rt/global/*
  - name: rq/global/*
  - name: rr/global/*

participants:
  - name: local
    kind: local
    domain: ${ROS_DOMAIN_ID}

  - name: global
    kind: local
    domain: 0

specs:
  discovery-trigger: any

I still went with the approach of one dediacted ROS_DOMAIN_ID per robot because that way I can optionally tune into a robot's internal communication by setting the ROS_DOMAIN_ID appropriately on a remote machine. Only down-side right now is still related to the previous post, the 5th point specifically: I cannot use both discovery-trigger: any and remove-unused-entities: true.

[Tuntenfisch]

During my testing I noticed that if I set discovery-trigger: reader and I attempt to do ros2 topic echo <topic published in another ROS_DOMAIN_ID but routed via the DDS router> the topic cannot be found unless I first create a subscriber that subscribes to this topic in the original ROS_DOMAIN_ID where that topic is published. I suppose this is because the DDS router doesn't create an internal reader/writer for routing the topic unless a reader is specifically created in the publishing ROS_DOMAIN_ID for a given topic. But if I solely want to publish a topic into the "shared" ROS_DOMAIN_ID without being interested in the published information in my own per-robot domain, I cannot do that. Or am I missing something?

Okay, the issue here has to do with a particularity of ros2 topic echo. When I use

specs:
  discovery-trigger: reader
  remove-unused-entities: true

as part of the DDS router config, set up a publisher

ros2 topic pub /global/test std_msgs/String 'data: Hello World'

in ROS_DOMAIN_ID=0, and an actual subscriber node in

import rclpy
from rclpy.node import Node
from std_msgs.msg import String


class SimpleSubscriber(Node):
    def __init__(self):
        super().__init__("simple_subscriber_node")

        self.subscription = self.create_subscription(
            String, "/global/test", self.listener_callback, 10
        )

    def listener_callback(self, msg):
        self.get_logger().info(f'Received: "{msg.data}"')


if __name__ == "__main__":
    rclpy.init()
    simple_subscriber = SimpleSubscriber()
    rclpy.spin(simple_subscriber)
    rclpy.shutdown()

in ROS_DOMAIN_ID=1, the DDS router ends up detecting the reader created by my SimpleSubscribernode and in turn creates a corresponding writer in ROS_DOMAIN_ID=1 to forward the topic.

But this doesn't happen if I just use ros2 topic echo /global/test, even when I opt to set the optional --spin-time parameter. My intuition was that ros2 topic echo basically creates a subscriber under the hood, so this is kind of surprising to me. Also ros2 topic list fails to see the topic (which I was expecting anyways). Obviously the ideal scenario would be if

specs:
  discovery-trigger: any
  remove-unused-entities: true

works. That way, ros2 topic utilities would work without issue and unused topics are automatically removed.

[Tuntenfisch]

Okay, I'm facing another issue now, which I want to illustrate with a simple example setup:

I have two DDS router instances running on the same machine. One bridges between domain 4 and domain 0, and the other one between domain 4 and domain 0. They are both in their separate docker containers, started using docker run -it --network=host <image> /bin/bash.

Bridge between domain 4 and domain 0

participants:
  - name: local
    kind: simple
    domain: 4

  - name: global
    kind: simple
    domain: 0

specs:
  discovery-trigger: any

Bridge between domain 3 and domain 0

participants:
  - name: local
    kind: simple
    domain: 3

  - name: global
    kind: simple
    domain: 0

specs:
  discovery-trigger: any

When I publish something in domain 4 with ROS_DOMAIN_ID=4 ros2 topic pub /topic std_msgs/String 'data: Hello World' and listen to it in domain 0 with ROS_DOMAIN_ID=0 ros2 topic echo /topic, I can see the published messages, indicating that the bridging between domain 4 and domain 0 is working. But when I instead try to listen to the published messages in domain 3 with ROS_DOMAIN_ID=3 ros2 topic echo /topic, I get nothing. Interestingly, ROS_DOMAIN_ID=3 ros2 topic list does show the topic /topic being discovered.

I've attached a video of my going through this example as well:

Screencast.from.21.04.2025.12.17.05.webm

The DDS router version used is

DDS Router v3.1.0
commit hash: b27a1acb5cf6ae65a8b26502f4c347f598ea1996

The ROS version is ROS2 Humble.

[Tuntenfisch]

I'm not entirely sure if this relates to the Warning posted in the DDS router documentation:

Do not configure two Participants in a way that they can communicate to each other (e.g. two Simple participants in the same domain). This will lead to an infinite feedback loop between each other.

[Tuntenfisch]

Okay, I'm facing another issue now, which I want to illustrate with a simple example setup:

I have two DDS router instances running on the same machine. One bridges between domain 4 and domain 0, and the other one between domain 4 and domain 0. They are both in their separate docker containers, started using docker run -it --network=host <image> /bin/bash.
Bridge between domain 4 and domain 0

Bridge between domain 3 and domain 0

When I publish something in domain 4 with ROS_DOMAIN_ID=4 ros2 topic pub /topic std_msgs/String 'data: Hello World' and listen to it in domain 0 with ROS_DOMAIN_ID=0 ros2 topic echo /topic, I can see the published messages, indicating that the bridging between domain 4 and domain 0 is working. But when I instead try to listen to the published messages in domain 3 with ROS_DOMAIN_ID=3 ros2 topic echo /topic, I get nothing. Interestingly, ROS_DOMAIN_ID=3 ros2 topic list does show the topic /topic being discovered.

I've attached a video of my going through this example as well:
Screencast.from.21.04.2025.12.17.05.webm

The DDS router version used is

DDS Router v3.1.0
commit hash: b27a1ac

The ROS version is ROS2 Humble.

Okay, this seems to be an issue with having both DDS router instances on the same host. When I run the containers with

# Create dedicated network instead of relying on host network.
docker network create dds_test_net

# Run first DDS Router container in terminal 1.
docker run -it \
  --hostname dds_router_4 \
  --network dds_test_net \
  <image> \
  /bin/bash

# Run second DDS Router container in terminal 2.
docker run -it \
  --hostname dds_router_3 \
  --network dds_test_net \
  <image> \
  /bin/bash

and keep the same respective bridge configs, bridging from domain 4 to domain 3 via domain 0 works. I'm not quite sure what the issue is with using the host network for all 3 participants (the 2 DDS routers and the ROS2 publish/subscriber). Would be great if someone more knowledgeable could explain this to me.

[juanlofer-eprosima]

Hi @Tuntenfisch , sorry for the delay.

Hi @Tuntenfisch,

1. Sure it could make sense in some scenarios.
2. It would be an option to achieve your goal I believe.
3. Probably, can't think of any at the moment. I suggest you use a default XML profile configuration in your robots using ignore participant flags, so that ROS 2 nodes only communicate in the same host. Then you could have a DDS-Router in each robot, with a local participant in a local domain (could be the same in all robots since communication is only local), adding also the corresponding ignore participant flag, and another participant in the global shared domain. By the way, you could use an allowlist rule like "/global/" instead.
4. See previous answer.
5. No there is no workaround. Why would you like to forward data if there is no subscriber to receive it?

Thank you for your feedback, regarding your points:

3. I think you are saying I could use the same ROS_DOMAIN_ID for each robot if I restrict the communication to local host only. This would effectively prevent me from getting any topic information (for example) from outside the robot because its communication never leaves the local host. But I suppose in certain scenarios it would be beneficial to have the ability to do something like `ROS_DOMAIN_ID=<specific robot domain ID> ros2 topic list` from a different machine to investigate a specific robots communication for debugging. Is my understanding of this limitation correct?

‎

5. During my testing I noticed that if I set `discovery-trigger: reader` and I attempt to do `ros2 topic echo <topic published in another ROS_DOMAIN_ID but routed via the DDS router>` the topic cannot be found unless I first create a subscriber that subscribes to this topic  in the original ROS_DOMAIN_ID where that topic is published. I suppose this is because the DDS router doesn't create an internal reader/writer for routing the topic unless a reader is specifically created in the publishing ROS_DOMAIN_ID for a given topic. But if I solely want to publish a topic into the "shared" ROS_DOMAIN_ID without being interested in the published information in my own per-robot domain, I cannot do that. Or am I missing something?

A slightly off topic question:

I'm using ROS2 Humble and was wondering which DDS router version i should be using. I think ROS2 Humble is using DDS v2.6.2 so I should probably choose the DDS router version compatible with this DDS version?

Also, the installation guide I found here mentions I can install using one of the following methods:

1. Colcon: I guess this would require me to then subsequently source the workspace before being able to use the router. It's also mentioned that if a DDS implementation is already installed, not all packages need to be built. Since ROS2 Humble is using Fast DDS I guess I wouldn't need to build Fast DDS again. But where does ROS2 Humble store it's DDS installation? Is `source /opt/ros/humble/setup.bash` sufficient?

2. CMake (local): I could potentially also skip building certain parts when I already have them installed via ROS2 Humble, correct? Is the procedure the same?

3. CMake (system-wide): Same question as for the local installation.

I suppose there is no right or wrong installation procedure to use. I'm setting up my software stack using a docker container so I would probably go with the system-wide installation because it doesn't require sourcing the workspace like the other two do.

This is quite a long post and I very much appreciate your help.

3. Yes that is the idea. And yes, that would be something you cannot do.
4. Your understanding is correct, by default information will not flow unless an external reader is discovered or discovery-trigger is changed.

[juanlofer-eprosima]

Okay, I'm facing another issue now, which I want to illustrate with a simple example setup:
I have two DDS router instances running on the same machine. One bridges between domain 4 and domain 0, and the other one between domain 4 and domain 0. They are both in their separate docker containers, started using docker run -it --network=host <image> /bin/bash.
Bridge between domain 4 and domain 0
Bridge between domain 3 and domain 0
When I publish something in domain 4 with ROS_DOMAIN_ID=4 ros2 topic pub /topic std_msgs/String 'data: Hello World' and listen to it in domain 0 with ROS_DOMAIN_ID=0 ros2 topic echo /topic, I can see the published messages, indicating that the bridging between domain 4 and domain 0 is working. But when I instead try to listen to the published messages in domain 3 with ROS_DOMAIN_ID=3 ros2 topic echo /topic, I get nothing. Interestingly, ROS_DOMAIN_ID=3 ros2 topic list does show the topic /topic being discovered.
I've attached a video of my going through this example as well:
Screencast.from.21.04.2025.12.17.05.webm
The DDS router version used is
DDS Router v3.1.0
commit hash: b27a1ac
The ROS version is ROS2 Humble.

Okay, this seems to be an issue with having both DDS router instances on the same host. When I run the containers with

Create dedicated network instead of relying on host network.

docker network create dds_test_net

Run first DDS Router container in terminal 1.

docker run -it
--hostname dds_router_4
--network dds_test_net

/bin/bash

Run second DDS Router container in terminal 2.

docker run -it
--hostname dds_router_3
--network dds_test_net

/bin/bash

and keep the same respective bridge configs, bridging from domain 4 to domain 3 via domain 0 works. I'm not quite sure what the issue is with using the host network for all 3 participants (the 2 DDS routers and the ROS2 publish/subscriber). Would be great if someone more knowledgeable could explain this to me.

This is probably a transport issue, because both believe they can communicate using shared memory transport but likely cannot access to the shared memory resource. Try adding "--ipc=host" to the docker run command. If that does not work, you could also try to add "transport: udp" to each participant configuration.