The mystery of the disappearing traceroute timestamp

Ever done a traceroute from one Cisco router to another and wondered why the second slot, on the final hop, shows an '*' instead of a timestamp like any of the above. Take for example the trace output below.

R1#traceroute 172.16.6.1 source GigabitEthernet 1.1
Type escape sequence to abort.
Tracing the route to 172.16.6.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.0.12.2 3 msec 1 msec 0 msec
  2 10.0.23.3 1 msec 1 msec 1 msec
  3 10.0.34.4 1 msec 1 msec 0 msec
  4 10.0.45.5 6 msec 1 msec 1 msec
  5 10.0.56.6 2 msec *  3 msec

Notice how only the final hop has a timeout (denoted by the '*') in the second portion - none of the previous hops had this.

Well, without spending as much time writing this post as I did finding out why this was, I will give a very short view of what causes this behaviour.

The way traces (in Cisco IOS) work is by sending a UDP packet to the destination host with each packet incrementing the destination port number as well as the TTL value. The packets will keep getting a "TTL exceeded in transit" reply from each hop until the TTL values is increased to the proper amount of hops it take to reach the destination network. When it finally has a TTL value high enough to reach a router without expiring the TTL, the destination router will respond with an ICMP type 3 (code 3) message - a destination unreachable (port unreachable) message.

Below is the packet capture from the router with the address 10.0.56.6.

Packet capture on R6

There are the three UDP packets sourced from inside interface of R1 (172.16.1.1) going to the inside interface of R6 (172.16.6.1), but there are only two ICMP unreachables sent in reply.

So, the problem is with an IOS default setting for ICMP unreachables. It can be found using the show run all command with little filtering (to parse out all the unrelated sections).

R6#show running-config all | include ip icmp
ip icmp rate-limit unreachable 500
ip icmp redirect subnet

The culprit is the ip icmp rate-limit unreachable 500 command. This setting can easily be tweaked to allow the traceroute to display correctly.

R6#configure terminal
R6(config)#no ip icmp rate-limit unreachable

Now the trace will display properly - so long as the destination is on R6.

R1#traceroute 172.16.6.1 source GigabitEthernet 1.1
Type escape sequence to abort.
Tracing the route to 172.16.6.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.0.12.2 3 msec 0 msec 0 msec
  2 10.0.23.3 1 msec 0 msec 1 msec
  3 10.0.34.4 1 msec 0 msec 1 msec
  4 10.0.45.5 1 msec 1 msec 1 msec
  5 10.0.56.6 2 msec 1 msec 2 msec

Now, you may wonder why it is only the final destination that has this issue and not every single hop. Well, only the final hop sends an ICMP unreachable - the rest send TTL expired.

Although the traceroute now displays properly, I must point out that the default setting is a default for a reason. You most likely do not wish to turn this off on production equipment unless you have a VERY good reason to; it prevents DoS attacks from trashing the router by having it respond with copious amounts of ICMP unreachable packets - so best leave it as is and just live with the one missing reply in the traces.

show run all to the rescue

This is gonna be a quick post!

Ever drawn a blank when trying to recall a specific command? You know what you're looking for and it's on the tip of your tongue. Well, I like to use the command show run all to look for the answer. If used right it can quickly find you the default of a command that would otherwise be omitted from regular show run command.

Of course, when looking for something specific, the command show run all won't help much, unless your objective is to wear down the spacebar on the keyboard. To find something specific it will have to be piped to either include some keywords or with the section command to filter out all the insane amounts of default commands.

Below is an example of the command used to show all the configured commands on interface FastEthernet2 on a Cisco 881 router. It lists all the commands configured on the interface that wouldn't be seen under normal circumstances.

AMBO-RT#sh run all | section interface FastEthernet2
interface FastEthernet2
 switchport access vlan 10
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 1
 switchport trunk allowed vlan 1-4094
 switchport mode access
 switchport voice vlan none
 switchport priority extend none
 switchport priority default 0
 mtu 1500
 no ip address
<further output omitted - it goes on for a while>

So, this command could be just the thing needed to jog your memory, when drawing a blank.

Also, if you like to read CLI output, like Joe Pantoliano reads the Matrix, go ahead and do a show run all and take it from the top - you will be amazed how many commands you never knew existed.

Configure Replace

If you plan on loading a lot of different topologies into your lab - be it physical or logical - you may want to consider learning the configure replace command to speed things up a bit. What this command can do for you in the lab is to cut down on the time it takes to "reset to zero" by having a base configuration for your router/switch, that you can then use to overwrite any changes done when tampering with different technologies or lab assignments. What I used to do was to make sure not to write anything to the config and then when I needed to reset the lab I would reload the devices and wait the excruciatingly long time it took for them to reload (this was with my hardware lab consisting of a couple of 2600 routers). When I started building a lab setup for my CCIE I knew I had to find a way to change configs faster and I found that this command would help me do just that.

This is how I setup my lab routers  (the routers I use are CSR1000v - Ciscos virtual cloud routers).

You boot up your router and configure the things you want to be configured just about always.

enable
configure terminal
!
hostname R1
!
logging buffered 8192
!
no aaa new-model
!
no ip domain lookup
!
no ip http server
no ip http secure-server
!
line con 0
 exec-timeout 0
 logging synchronous
!
end

When you are done setting up the very basics  - save the running-config to a file on flash.

copy running-config flash:/config/base.conf

Now, whenever you have configured anything on the router - like, say, some DMVPN or EIGRP configuration - and you want to reload it to the base to start another lab, this is what you do.

From the privileged exec mode enter the following command

R1#configure replace flash:/config/base.conf force
Total number of passes: 1
Rollback Done
R1#
*Nov 12 10:30:42.186: Rollback:Acquired Configuration lock.
R1#

This makes the running-config identical that of the base.conf file saved earlier. The "Total number of passes: 1" indicates it took 1 pass of the config to make it identical. The amounts of passes it will take depends on how much the runnin-config and the base.conf file differs.

There is a bit more to the command if you want to use it outside of the lab, but this just about covers what you may want in a lab environment.

vlan internal allocation policy ascending

If you have worked with Cisco switches for some time you may have noticed this command when viewing the configuration. Ever wonder what it actually means?

Well, I have and today I just so happened to satisfy my own curiosity by looking it up.

In short, it means that the switch will allocate VLANS for internal use starting from VLAN 1006 (because VLAN 1002-1005 are reserved for some legacy stuff that nobody dares remove even though the calendar says 2014).

This may make you go "Oh. Well, of course, that's what it means..." and think no more of it. But, if you are anything like me, you probably went "Ok. What does that mean?". What are VLANS used for "internally" on a Cisco switch?

One example that is easy to setup in the lab is this:

SW01(config)#int gi0/33
SW01(config-if)#no switchport
SW01(config-if)#ip address 192.168.1.1 255.255.255.0

We just changed a switchport on a Catalyst switch into a routed port. Another way of accomplishing this goal would be to do this:

SW01(config)#int gi0/33
SW01(config-if)#switchport access vlan 10
% Access VLAN does not exist. Creating vlan 10
SW01(config-if)#int vlan 10
SW01(config-if)#ip address 192.168.1.1 255.255.255.0

Assuming that interface gi0/33 is the only interface in VLAN 10 - this would pretty much accomplish what the no switchport command does. In fact, it is exactly what the no switchport command does and the vlan internal is the key to show it.

Let us look at the no switchport scenario and see if we can't peek a little behind the curtains.

SW01#show ip interface brief | include GigabitEthernet0/33
GigabitEthernet0/33    192.168.1.1     YES manual down                  down

So, the interface has an IP address assigned manually and it is currently in the down-down state (no cable is attached). Let us try to see some of the internal vlan usage.

SW01#sh vlan internal usage
VLAN Usage
---- --------------------
1006 GigabitEthernet0/33

This shows us that VLAN 1006 has been assigned to GigabitEthernet0/33 (per the allocation policy).

So, what this means is, that when you issue the no switchport command, the switch creates VLAN 1006 and assigns it the IP address and chains it to the physical interface.

A thing to keep in mind is that internally assigned VLANs and user created VLANs share the same database so an error message like the one below is shown if I try to manually create a VLAN that is already assigned internally.

SW01(config)#vlan 1006
SW01(config-vlan)#exit
% Failed to create VLANs 1006
VLAN(s) not available in Port Manager.
%Failed to commit extended VLAN(s) changes.
*Jun 10 08:20:11.932: %PM-4-EXT_VLAN_INUSE: VLAN 1006 currently in use by GigabitEthernet0/33
*Jun 10 08:20:11.932: %SW_VLAN-4-VLAN_CREATE_FAIL: Failed to create VLANs 1006: VLAN(s) not available in Port Manager

You can probably imagine some of the scenarios that might arise if you don't know that this is the way Catalyst switches accomplishes this - maybe you have your own tale to tell on the subject.
Here is a link to an article about one of the more likely situation you might learn of this "feature" in the IOS: http://packetpushers.net/cisco-internal-vlan-usage/

Edit (27-10-2014):
There are a few more things that the no switchport command disables.

You can mimic the no switchport a little better if you issued the following commands:

SW01(config)#no spanning-tree vlan 10
SW01(config)#no mac address-table learning vlan 10
SW01(config)#interface gi0/33
SW01(config-if)#switchport mode access
SW01(config-if)#switchport access vlan 10
SW01(config-if)#switchport nonegotiate
SW01(config-if)#no vtp
SW01(config-if)#exit
SW01(config)#interface vlan 10
SW01(config-if)#ip address 192.168.1.1 255.255.255.0

This disables spanning-tree and mac learning on the vlan, adds the port to the vlan as an access port, disables DTP negotiation and VTP on the port and finally configures an IP address on the SVI (switched virtual interface).