SlicerROS2_specifics.md

The high level ROS functionalities (publisher, subscriber, Tf2 broadcast, Tf2 lookup or parameter client) are all encapsulated as Slicer MRML nodes, i.e. derived from the class vtkMRMLNode and added to the MRML Scene. We ultimately decided to implement the SlicerROS2 module in C++ and rely on the VTK/Slicer build to provide the Python bindings. The SlicerROS2 module is compiled against the Slicer and ROS2 libraries which is a bit challenging since both packages have their own CMake macros and have some Python version requirements. The CMakeLists.txt provided along SlicerROS2 works but you’ll have to ignore some error and warning messages.

One of the main challenges of integrating ROS (Robot Operating System) into 3D Slicer is determining how the two systems should run together — in other words, figuring out the execution model. ROS is designed around a network of asynchronous nodes that continuously exchange messages through their own looping process, while Slicer is an interactive, event-driven application built on Qt and VTK, which already runs its own main GUI loop. Because both frameworks expect to “own” the program’s flow of control, developers must decide how ROS will operate within Slicer without freezing the interface or causing thread conflicts. This typically involves defining when and where ROS code executes — for example, running ROS in a background thread, polling it with timers, or connecting it through an external bridge — and ensuring that all updates to the Slicer scene and UI happen safely on the main thread. In short, the “execution model” question is about harmonizing ROS’s real-time message-driven behavior with Slicer’s single-threaded, event-driven architecture.

One of the challenges of integrating ROS in Slicer is to figure out the execution model. ROS relies heavily on callbacks triggered by external messages. This requires to either use a separate thread to “spin” the ROS event loop or periodically call the ROS “spin” method from the application’s main thread. Since we are heavily relying on the MRML scene, using a separate thread is not trivial. Therefore the SlicerROS2 module relies on the main Slicer thread to trigger a periodic call to the ROS spin. We currently use a Qt timer to trigger this periodic call.

The two packages also differ in their design patterns. Slicer (and VTK) strongly relies on base classes and inheritance to allow runtime decisions. Meanwhile, ROS heavily relies on templates and type traits which are handled at compilation time. Most ROS communication mechanisms only support a finite number of data types (e.g. parameters are booleans, integers, floating points, strings or vector of aforementioned types, tf2 uses transforms only…) so this is not a major issue.

The main difficulty lies in supporting many ROS topics and services For our code, we ended up using templates for our internal data structures and add some macros to generate the vtk user classes. These macros use template specialization and add some methods to create a C++ class that can be used within Slicer (including the Python bindings generation).

We added a code generator that can create a VTK object mimicking a ROS message. The code generator will also create the overloaded conversion methods (vtkSlicerToROS2 and vtkROS2ToSlicer) as well as code to call all the macros required to create the publisher or subscriber node. The macro generate_ros2_nodes is used in MRML/CMakeLists.txt and one can add new ROS messages types. New publisher and subscriber nodes will be available after the SlicerROS2 module is recompiled.

generate_ros2_nodes( GENERATED_FILES_PREFIX "SLICER_ROS2_GENERATED" PUBLISHERS "geometry_msgs/msg/PoseStamped" "geometry_msgs/msg/TransformStamped" "geometry_msgs/msg/WrenchStamped" "sensor_msgs/msg/Joy" "sensor_msgs/msg/JointState" "geometry_msgs/msg/PoseArray" SUBSCRIBERS "geometry_msgs/msg/PoseStamped" "geometry_msgs/msg/TransformStamped" "geometry_msgs/msg/WrenchStamped" "sensor_msgs/msg/Joy" "sensor_msgs/msg/JointState" "geometry_msgs/msg/PoseArray" DEPENDENCIES "std_msgs/msg/Header" "builtin_interfaces/msg/Time" )

The CMake macro generate_ros2_nodes allows users to quickly add new publishers and subscribers.

If you are adding messages from a ROS package not already used by SlicerROS2, you might have to edit the main CMakeLists.txt and find_package for the new ROS package. The SlicerROS2 module will automatically convert between the default 3D frames conventions in Slicer and ROS. Slicer (and by extension all VTK objects in Slicer) follow the RAS convention and distances are provided in millimeters. Meanwhile uses the RHS convention and SI units (meters).

The SlicerROS2 module always creates a default ROS node (internally a rclcpp::node). This node is both a ROS node and a MRML node, hence the unfortunate name vtkMRMLROS2NodeNode. This node is added to the MRML scene and should be used to add your custom vtkMRMLROS2 nodes (topics, parameter…). It is possible to add more ROS nodes in SlicerROS2 but this feature has not been tested extensively for the first release.

To retrieve the default ROS node: rosLogic = slicer.util.getModuleLogic('ROS2') rosNode = rosLogic.GetDefaultROS2Node()

Key types: Slicer type --> ROS type --> SlicerROS2 "name" vtkTransformCollection --> geomtry_msgs::msg::PoseArray --> PoseArray vtkPoints --> sensor_msgs::msg::PointCloud --> PointCloud

Example SlicerROS2 node name: vtkMRMLROSPublisherPoseStampedNode

Supported publishers and subscribers: rosLogic = slicer.util.getModuleLogic('ROS2') rosNode = rosLogic.GetDefaultROS2Node() rosNode.RegisteredROS2PublisherNodes() rosNode.RegisteredROS2SubscriberNodes()

Publishers To create a new publisher, one should use the MRML ROS2 Node method vtkMRMLROS2NodeNode::CreateAndAddPublisherNode.

This method takes two parameters: the class (type) of publisher to be used (full or short name). Publishers are provided for most common data types (from Slicer to ROS messages). The full list can be found in the Slicer ROS logic class (Logic/vtkSlicerROS2Logic.cxx) in the method RegisterNodes. The topic name (e.g. std::string)

Publishers ar etriggered by calling the Publish method:

rosLogic = slicer.util.getModuleLogic('ROS2') rosNode = rosLogic.GetDefaultROS2Node()

optional, shows which publishers are available

rosNode.RegisteredROS2PublisherNodes()

example with full class name

pubString = rosNode.CreateAndAddPublisherNode('vtkMRMLROS2PublisherStringNode', '/my_string')

run ros2 topic echo /my_string in a terminal to see the output

pubString.Publish('my first string')

example with short class name, Pose will be expended to vtkMRMLROS2PublisherPoseNode

pubMatrix = rosNode.CreateAndAddPublisherNode('Pose', '/my_matrix')

run ros2 topic echo /my_matrix in a terminal to see the output

mat = pubMatrix.GetBlankMessage() # returns a vtkMatrix4x4 mat.SetElement(0, 3, 3.1415) # Modify the matrix so we can see something changing pubMatrix.Publish(mat)

To remove the publisher node, use vtkMRMLROS2NodeNode::RemoveAndDeletePublisherNode. This method taks the topic name (std::string)

Subscribers To create a new subscribre, uses the MRML ROS2 Node method vtkMRMLROS2NodeNode::CreateAndAddSubscriberNode

This method takes two parameters: the class (type) of subscriber to be used (full or short name). Subscribers are provided for most common data types (from Slicer to ROS messages). The full list can be found in the Slicer ROS logic class (Logic/vtkSlicerROS2Logic.cxx) in the method RegisterNodes. The topic name (e.g. std::string)

Subscriber nodes get updated when the ROS2 node is spun. Users can set their own callback to act on newly received messages using an observer on the MRML ROS subscriber node. The ;ast message received can be retrieved using GetLastMessage.

rosLogic = slicer.util.getModuleLogic('ROS2') rosNode = rosLogic.GetDefaultROS2Node()

optional, shows which subscribers are available

rosNode.RegisteredROS2SubscriberNodes() subShape = rosNode.CreateAndAddSubscriberNode('geometry_msgs/msg/PoseArray, '/needle/state/current_shape')

run ros2 topic pub /my_string in a terminal to send a string to Slicer

m_shape = subShape.GetLastMessage()

alternate, get a string with the full message

m_shape_yaml = subShape.GetLastMessageYAML()

since the subscriber is a MRML node,

you can also create an observer (callback)

to trigger some code when a new message is received.

example callback function:

def myCallback(caller=None, event=None): message = subShape.GetLastMessage() print("Last message received by subscriber: {}.".format(message))

add the observer with callback

observerId = subShape.AddObserver('ModifiedEvent', myCallback)

the last message received will print in the python console

in Slicer when data is published to /my_shape

another example - updating transforms based on subscribed pose data

(ie. for an optical tracker with a ros wrapper)

subPose = rosNode.CreateAndAddSubscriberNode('Pose', '/StylusToTracker') transform = slicer.mrmlScene.AddNewNodeByClass('vtkMRMLLinearTransformNode', 'StylusToTracker')

define the callback

def updateTransforms(caller=None, event=None): pose = subPose.GetLastMessage() transform.SetMatrixTransformToParent(pose) print("Last message received by subscriber: {}.".format(message))

add the observer with callback

observerId = subPose.AddObserver('ModifiedEvent', updateTransforms)

To remove the subscriber node, use the method vtkMRMLROS2NodeNode::RemoveAndDeleteSubscriberNode. This method taks the topic name (std::string).

How to find a Python function for any Slicer features All features of Slicer are available via Python scripts. Slicer script repository contains examples for the most commonly used features.

To find out what Python commands correspond to a feature that is visible on the graphical user interface, search in Slicer’s source code where that text occurs, find the corresponding widget or action name, then search for that widget or action name in the source code to find out what commands it triggers.

Complete example: How to emulate selection of FOV, spacing match Volumes checkbox in the slice view controller menu?

Go to Slicer project repository on github

Enter text that you see on the GUI near the function that you want to use. In this case, enter "FOV, spacing match Volumes" (adding quotes around the text makes sure it finds that exact text)

Usually the text is found in a .ui file, in this case it is in qMRMLSliceControllerWidget.ui, open the file

Find the text in the page, and look up what is the name of the widget or action that it is associated with - in this case it is an action named actionSliceModelModeVolumes

Search for that widget or action name in the repository, you should find a source file(s) that use it. In this case it will is qMRMLSliceControllerWidget.cxx

Search for the action/widget name, and you’ll find what it does - in this case it calls setSliceModelModeVolumes method, which calls this->setSliceModelMode(vtkMRMLSliceNode::SliceResolutionMatchVolumes), which then calls d->MRMLSliceNode->SetSliceResolutionMode(mode)

This means that this action calls someSliceNode->SetSliceResolutionMode(vtkMRMLSliceNode::SliceResolutionMatchVolumes) in Python syntax it is someSliceNode.SetSliceResolutionMode(slicer.vtkMRMLSliceNode.SliceResolutionMatchVolumes). For example, for the red slice node this will be:

sliceNode = slicer.mrmlScene.GetNodeByID('vtkMRMLSliceNodeRed') sliceNode.SetSliceResolutionMode(slicer.vtkMRMLSliceNode.SliceResolutionMatchVolumes)