Accurate Sensor Timing in ROS 2

It is crucial to obtain correct and accurate timing in autonomous robotic systems.

Complex perception, localisation and planning algorithms rely on combining, or ‘fusing’, data from a multitude of different sensors, which very likely run at different frequencies and out of sync of each other. The sensor data must be timestamped so that their relation in time can be accurately determined. If there is any error in the timestamps, at the very start of the robot’s sense-think-act loop, it will make judgement errors that may be hard if not impossible to recover from. A prime example of ‘garbage in, garbage out’.

Recently I set out to refine the time-stamping in the ROS 2 V4L2 camera package that I maintain. Not satisfied with some of the simple solutions that several other drivers use, I wanted it to be as accurate as possible in all camera and ROS 2 configurations, for all use cases. This resulted in two changes, which addressed both absolute and relative timing.

Why did I need a total of 42 lines to implement something as seemingly trivial as V4L2Camera::determineStamp?

And why is the node now publishing messages of type sensor_msgs::msg::TimeReference that you may never have heard of before?

The Problem

It can be difficult to determine the exact time of the event that a piece of sensor data represents—its timestamp. One can easily capture the time that the data arrived in some part of the data processing pipeline, such as the sensor driver node in a ROS 2 system. However, there will always be lag between the event and the time at which the data about it reaches some point of computation.

Sensors may come with their internal clock, so that they can set a timestamp as close as possible to the actual measurement of the event. The important problem to solve then is how to synchronise that clock with the clock on the computer that runs your robotics software stack. The most precise way to do so is to use the Precision Time Protocol (PTP). Unfortunately not all sensors support PTP, and you also must make sure that your networking hardware supports it, including any managed switch and NIC between the sensor and your compute.

Yet other factors affecting sensor timing include:

• Your computer has different clocks, and your software may use any as its reference.
• Data collection is often not instantaneous. For instance, points at different headings in a pointcloud collected by a spinning lidar have different timestamps, simply because it takes time to make a full rotation. Similarly, a camera with a rolling shutter measures pixel values at different times.
• Time may jump, when a service on your robot’s computer determines it needs to fix a wrong clock, or some international body decides to throw in an extra second because too much of the Earth’s ice caps has melted.

Luckily we can limit our scope here to times in V4L2 and ROS 2.

Computer Clocks

Before looking at timestamps as handled by V4L2 and by ROS 2, we need to learn a little bit about the different clocks on your computer. Because your computer keeps track of time in different ways, and each way defines a different ‘clock’. One of the main distinctions between these clocks is whether they keep track of ‘real’ time, or ‘monotonic’ time.

‘Real’ time is likely the one you are most familiar with: this is the time that underpins what is shown on your desktop, or the time that is printed when you run date. Note that it is not related to the concept of ‘real-time computing’. Real time is also sometimes called ‘system time’, and on Linux you can retrieve this time using a call to clock_gettime(CLOCK_REALTIME, &tp).

Although real time is what humans mostly use, it has a big potential downside when using it in computation: it can jump. Such jumps can happen due to things like leap seconds, somebody manually setting the clock on a machine, or corrections by something like NTP of such an erroneous manual change or of drift of your computer’s clock. This means that the difference between two timestamps from such a clock may not be close to the actual time passed between taking those timestamps. Most strikingly, real time can jump backwards, so that the measured time delta between two timestamps can become negative. Such jumps are hardly common, but can seriously mess with algorithms that depend on the time between measurements, such as sensor fusion, localisation and SLAM.

This is why monotonic time exists: a monotonic clock is not affected by discontinuous jumps and will never go backwards. On Linux there are several slightly different monotonic clocks:

Think of a ‘real’ time clock as your alarm clock ⏰, and of a ‘monotonic’ clock as a stopwatch ⏱️.

Timestamps in V4L2

Video4Linux2 (V4L2) provides an extensive API for capturing video data. In the most typical streaming I/O usage data is exchanged through buffers. A v4l2_buffer structure is used to pass pointers to and metadata about the buffers holding video data. That structure has a struct timeval timestamp field. We can grab that, and case closed!

…if we didn’t have to worry about what clock is being used in which part of the system. The v4l2_buffer structure also has a flags field that captures many details about the captured data. The most relevant flags are under the V4L2_BUF_FLAG_TIMESTAMP_MASK mask which gives the following options:

Some observations from this:

• Linux 3.9 was released on April 28, 2013. It is highly unlikely any advanced robotics system uses a kernel that old, so we should expect no driver to still use V4L2_BUF_FLAG_TIMESTAMP_UNKNOWN. However, a badly written driver is not unheard of, so we will still handle this case.

• When we encounter V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC, we know the exact reference clock: CLOCK_MONOTONIC. This means that the timestamps are subject to tiny incremental adjustments, however those are mostly meant to counteract drift and will never be backwards, so they should be beneficial.

  Also note that the timestamps do not include time during which a system is suspended. This should only be an issue when comparing timestamps from before and after the suspension; not likely to be too disruptive, but something to be aware of.

• We will ignore mem2mem devices, which are hardware video codecs and processing devices.

Another potentially interesting flag is V4L2_BUF_FLAG_TSTAMP_SRC_MASK, which tells us whether the timestamp has been taken when the last pixel of the frame has been received/transmitted (end-of-frame), or when the exposure of the frame has begun (start-of-exposure). This may be helpful to refine per-pixel latency calculations, however in practice I have only encountered end-of-frame, and there is no clean way to publish this information using standard ROS message types anyway.

Timestamps in ROS 2

Many ROS 2 messages have a header with a stamp field that should contain the moment in time that the data in the message is associated with. You could use standard functions like clock_gettime(), or std::chrono in C++, to set those timestamps, however ROS 2 provides its own abstraction on top of these. This is done in order to support simulating the progression of time, during log replay, including jumping back and forth in time and pausing, as well as simulations that could run faster or slower than real time, and also could be paused. A detailed discussion of the approach taken in ROS 2 can be found in the Clock and Time design document.

There are 3 clock types in ROS 2:

• RCL_SYSTEM_TIME - equivalent to std::chrono::system_clock, which in practice is equivalent to CLOCK_REALTIME.
• RCL_STEADY_TIME - equivalent to std::chrono::steady_clock, which in practice is equivalent to CLOCK_MONOTONIC on Linux, but CLOCK_MONOTONIC_RAW on macOS.
• RCL_ROS_TIME - equivalent to RCL_SYSTEM_TIME when no ‘ROS Time Source’ is active, the latest value reported by such a source otherwise.

A ‘ROS Time Source’ is any source that publishes on the /clock topic, and is active for a node if the use_sim_time parameter is set on it. Typically this is published by either ros2 bag play, or a simulator like Gazebo.

Technically there is also RCL_CLOCK_UNINITIALIZED, which indicates no clock time is available yet. We will handle this to follow proper defensive coding principles, though it should not be encountered once a node is running.

Any timestamps shared between nodes should use ROS time, so that the full system is consistent with the main clock in use. The default clock used by a node can be configured by setting the clock_type on a node’s NodeOptions, but in practice there is not often a need to do so.

Handling Different Clock Combinations

When developing a ROS 2 sensor driver package you are working on the edge of the ROS 2 system. It is likely that you have to deal with timers and clocks that are unaware of and independent of ROS time. This is indeed the case for the V4L2 camera driver: V4L2 provides its own buffer timestamps, but the driver node should publish messages in ROS time. Because each could use different clocks under the hood, we in theory have to handle the full Cartesian product of all possible combinations. In practice it comes down to the following:

The last combination should by far be the most common; let’s break it down into its 2 branches:

1. RCL_ROS_TIME == RCL_SYSTEM_TIME

The monotonic clock generally started counting when you turned on the robot, whereas the system clock started counting since the Unix epoch, which is 00:00:00 UTC on 1 January 1970. So to convert from monotonic time to system time we have to add some delta.

One could measure the difference between CLOCK_REALTIME and CLOCK_MONOTONIC at the start of the node once, and add that to the V4L2 buffer time at every frame. This is cheap and is what some ROS drivers do, however this approach is susceptible to jumps between the real time and monotonic clocks, induced by e.g. NTP.

Instead, we can measure the difference at every frame: every time a new buffer is received (dequeued) from V4L2, we sample CLOCK_REALTIME and CLOCK_MONOTONIC and add the difference between them to the timestamp read from the buffer. So the image timestamp becomes:

stamp = buf.timestamp + (CLOCK_REALTIME - CLOCK_MONOTONIC)
      = CLOCK_REALTIME - (CLOCK_MONOTONIC - buf.timestamp)

The second form shows that this approach is equal to measuring the dequeue latency CLOCK_MONOTONIC - buf.timestamp (the time duration between when the buffer timestamp was set by V4L2 and when the ROS node obtained the buffer) and subtracting that from the real time observed after dequeuing the buffer, which makes sense.

2. RCL_ROS_TIME != RCL_SYSTEM_TIME

If get_clock()->ros_time_is_active() returns true, there is an external time source publishing on /clock, the current node has use_sim_time set, and it has received a time message on /clock. This is not really a common situation for a sensor driver to be in. You wouldn’t normally run a hardware driver while replaying a log or when running a simulation; you would either publish the recorded outputs of the driver or run a simulated version of it. Yet, it is a valid scenario, possibly in some Hardware-in-the-loop (HIL) setups, so let’s still try to handle it as well as possible.

It is most likely that in this situation one would play a log or run a simulation with a real time factor of 1, i.e. not sped up or slowed down. Otherwise the camera would run at an unrealistic pace compared to the rest of the system. Under this assumption we can use ROS time, retrieved with Node::now(), in place of real time.

To enhance the realism of the timestamps, we can also subtract the dequeue latency in this case, so that subscribers measure a data delay similar to what they would measure in a live system.

Accurate Inter-Frame Duration with TimeReference

The approach above gives absolute timestamps as close as possible to the real time that a camera frame is associated with. But, we may have lost precision when we are interested in the exact time that passed between two frames. This can be detrimental for some algorithms that rely on this time delta, like any method that estimates velocity from camera data.

The cause of this drift is that the measurements of CLOCK_REALTIME and CLOCK_MONOTONIC are not done at exactly the same time, neither of each other nor of when the camera frame is dequeued. Any CPU scheduling delay of these instructions or CPU interrupts can add jitter to these measurements, which compounds when we perform the arithmetic to calculate the final message timestamp, and even further when we take the difference of the timestamps of different frames with different amounts of jitter.

In practice, the error introduced this way into the time difference is very small; on my machine I measured it to be up to a few dozen nanoseconds. But we said we want the most accurate timing possible, and luckily it is easy to supply a way to also achieve that for relative calculations.

The most accurate delta available to us is the difference between the buffer timestamps supplied by V4L2, but we have lost direct access to those by converting them to ROS time. The method of determining the latency between CLOCK_REALTIME and CLOCK_MONOTONIC only once at startup would still have allowed for calculating the original delta, because the fixed latency offset would be cancelled out. However, we already found out that that method is bad for absolute timing.

In the ROS 2 V4L2 Camera node this is now solved by publishing a little bit of extra information on top of the image messages: it now also publishes a sensor_msgs/msg/TimeReference. Its definition is as follows:

# Measurement from an external time source not actively synchronized with the system clock.

std_msgs/Header header            # stamp is system time for which measurement was valid
                                  # frame_id is not used

builtin_interfaces/Time time_ref  # corresponding time from this external source
string source                     # (optional) name of time source

We indeed have an external time source: the V4L2 buffer timestamps. By setting the header timestamp to the same value as that of the image message, and setting time_ref to the value received via buf.timestamp, we allow downstream consumers to use the most accurate absolute time we could devise via the image timestamps, but also to recover the raw buffer timestamps if desired, to calculate the most accurate time deltas with.

You would typically use a message_filters::TimeSynchronizer together with an image_transport::SubscriberFilter to subscribe to both the image and the time reference topics in order to receive their corresponding messages together.

Conclusion

All of the above should mean that the V4L2 Camera driver now provides the best timestamps possible.

Having said that, the actual achievable accuracy is probably most often limited by the cameras used with this driver, which are commonly consumer grade USB cameras, or cameras attached to simple SBCs such as Raspberry Pis (although the package supports any camera that supports V4L2). As mentioned previously, when the highest time accuracy is required, it is better to invest in PTP-capable hardware.

Either way I think it is important to understand how clocks and time-stamping works in computers and robotic systems in general, since getting them right is crucial for getting the best results out of your downstream algorithms that indubitably will depend on proper timing. I hope this post has contributed to that understanding.

Finally, if you find a bug, or know of a way to improve time stamps even further, feel free to open a ticket on the ros2_v4l2_camera repository!