The mystery of the disappearing traceroute timestamp

Ever done a traceroute from one Cisco router to another and wondered why the second slot, on the final hop, shows an '*' instead of a timestamp like any of the above. Take for example the trace output below.

R1#traceroute 172.16.6.1 source GigabitEthernet 1.1
Type escape sequence to abort.
Tracing the route to 172.16.6.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.0.12.2 3 msec 1 msec 0 msec
  2 10.0.23.3 1 msec 1 msec 1 msec
  3 10.0.34.4 1 msec 1 msec 0 msec
  4 10.0.45.5 6 msec 1 msec 1 msec
  5 10.0.56.6 2 msec *  3 msec

Notice how only the final hop has a timeout (denoted by the '*') in the second portion - none of the previous hops had this.

Well, without spending as much time writing this post as I did finding out why this was, I will give a very short view of what causes this behaviour.

The way traces (in Cisco IOS) work is by sending a UDP packet to the destination host with each packet incrementing the destination port number as well as the TTL value. The packets will keep getting a "TTL exceeded in transit" reply from each hop until the TTL values is increased to the proper amount of hops it take to reach the destination network. When it finally has a TTL value high enough to reach a router without expiring the TTL, the destination router will respond with an ICMP type 3 (code 3) message - a destination unreachable (port unreachable) message.

Below is the packet capture from the router with the address 10.0.56.6.

Packet capture on R6

There are the three UDP packets sourced from inside interface of R1 (172.16.1.1) going to the inside interface of R6 (172.16.6.1), but there are only two ICMP unreachables sent in reply.

So, the problem is with an IOS default setting for ICMP unreachables. It can be found using the show run all command with little filtering (to parse out all the unrelated sections).

R6#show running-config all | include ip icmp
ip icmp rate-limit unreachable 500
ip icmp redirect subnet

The culprit is the ip icmp rate-limit unreachable 500 command. This setting can easily be tweaked to allow the traceroute to display correctly.

R6#configure terminal
R6(config)#no ip icmp rate-limit unreachable

Now the trace will display properly - so long as the destination is on R6.

R1#traceroute 172.16.6.1 source GigabitEthernet 1.1
Type escape sequence to abort.
Tracing the route to 172.16.6.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.0.12.2 3 msec 0 msec 0 msec
  2 10.0.23.3 1 msec 0 msec 1 msec
  3 10.0.34.4 1 msec 0 msec 1 msec
  4 10.0.45.5 1 msec 1 msec 1 msec
  5 10.0.56.6 2 msec 1 msec 2 msec

Now, you may wonder why it is only the final destination that has this issue and not every single hop. Well, only the final hop sends an ICMP unreachable - the rest send TTL expired.

Although the traceroute now displays properly, I must point out that the default setting is a default for a reason. You most likely do not wish to turn this off on production equipment unless you have a VERY good reason to; it prevents DoS attacks from trashing the router by having it respond with copious amounts of ICMP unreachable packets - so best leave it as is and just live with the one missing reply in the traces.