WAN T1 Overview
For a network engineer or systems administrator confronting the WAN links in their network, three significant differences between WANs and LANs are immediately apparent.
WAN troubleshooting is a team effort
First, and in some ways the most important difference, the WAN is not under your sole control. Cooperation with others outside your company is a necessity.
You may lease a circuit connecting two sites within your own company, or you may lease a WAN connection to a service provider such as an ISP. Even if both ends of the circuit are "yours," the common carrier (typically a telephone company) owns, operates, and maintains the intervening lines, with all their attendant switches, repeaters, conditioners, and so on.
Maintaining the performance of your WAN connection is a team effort. Those things that are within your own control, you will of course learn to manage. But it is equally important to be able to provide the right information to the common carrier and to the "other end" of your connection when a problem requires action by others.
Analog makes a difference
Second, the analog nature of the WAN connection is important. While it is beyond the scope of this manual, a careful look at the analog aspects of WAN connectivity is always in order when troubleshooting. Long runs from the Telco demarc (an extended Demarc), line build out (voltage adjustments to optimize signal clarity) and a number of other factors have a significant impact on your WAN connection.
Another reminder of the analog nature of WANs is the way in which they can slowly degrade over time. Keeping track of throughput, error rates and performance over extended periods of time is a good practice. The records you keep may help you not only to identify problems, but also to make your case with the Telco engineers, and to help them diagnose-and fix-the problem.
WAN protocols share a different legacy
Finally, WAN protocols are the predecessors of Ethernet and IP, and they evolved in a very different environment with very different ideas of perfection. Two ideas (deprecated or totally rejected by the designers of Ethernet and IP) have shaped WAN protocols: very high reliability (founded on extremely reliable hardware) and a topology of switched point-to-point connections.
The emphasis on reliability is understandable. A failure rate of 1 in 100,000 seems trivial with only 25 nodes to maintain. With tens and hundreds of millions of nodes, it becomes a nightmare. The corollary of this real need for near-perfection was a glacial pace of change. Exhaustive testing, a consensus approach to innovation, and a reluctance to discard anything that actually works, have all characterized traditional WAN developments.
The topology of the switched network that creates and tears down a unique end-to-end path between any two communicating nodes is more than just a legacy of the traditional telephone system. It has also been a principle part of the business strategy of the carriers.
The various physical specifications of T1 and E1 lines and the link layer WAN protocols that use them are all influenced by these factors.
Figure H.1 OSI 7-layer model, showing WAN and IEEE protocols
Physical aspects of T1/E1
The voice communications requirements of the telephone network helped define the standards of the telecommunications industry. The physics of electrical signalling to create a full-duplex voice connection led to the world-wide adoption of 64 kbps as the standard for a single telephone "line," now called a DS0 (Digital Signal zero).
Individual lines were bundled together and later multiplexed together to form larger units and higher bandwidth. For example, 24 DS0 lines are bundled together to form a T1, two T1's to form a T1c, two of those to form a T2 (= 4 x T1), and seven T2's to form a T3 (with 672 DS0s).
Although the 64Kbps bandwidth of a DS0 is a worldwide standard, the exact sizes of the bundles differ slightly among the three main "spheres of influence" carved out by telephone monopolies in North America (T), Europe (E), and Japan (J). For example, E1 = 30 x DS0, E2 = 4 x E1, (but 3 x E1 is approximately = T2), E3 = 4 x E2, and E4 = 4 x E3 (or 3 x T3). The J1, J1c and J2 are identical to the T carriers of the same number, but J3 = 5 x J2, and J4 = 3 x J3. The hierarchy as a whole is known as the "Plesiochronous Digital Hierarchy" (PDH).
Table H.1 North American PDH (partial) Carrier data rate DS0s DS0 0.064Mbps 1 DS1 1.544Mbps 24 DS2 6.176Mbps 96 DS3 44.736Mbps 672 DS4 274.176Mbps 4032
T1 and E1 are roughly the same class of connection in terms of available bandwidth. The T1 line rate is 1.544 Mbps, which consists of 24 DS0 (1.536 Mbps) plus signalling. The line rate for E1 is 2.048 Mbps, with no room left over for signalling. Instead, the first channel (timeslot 0, in E1) is used for framing information. In some E1 framing schemes, channel 16 is also used for signalling. On an E1 line using PCM-30 framing, for example, channels 1-15 and 17-31 are available for user data, and these 30 DS0 provide 1.920 Mbps for data.
T1 signalling and framing
T1/E1 lines, whether whole or fractional, use time division multiplexing (TDM) to send multiple channels over a single pair of wires (one pair in each direction). Each DS0 has its own time slot.
T1 is a synchronous communications medium and accurate timing is absolutely crucial to performance. Devices take their timing information from the network.
At the most rudimentary level, each time slot in a T1 line allows the transmission of 192 bits, plus a framing bit at the beginning of each time slot or "frame." The word "frame" is in quotes here because this "slug of data filling a time slot" bears little or no resemblance to anything that would be recognized as a packet or frame on a LAN. It has one bit and a very precise synchronization between the sender and receiver to distinguish it from the rest of the voltage pulses on the line. When a connection is first made, it can take a moment to establish the framing!
More sophisticated framing schemes introduced in the 1970s and '80s allowed the condition of the link to be monitored. The first such scheme was the Superframe (SF) which groups 12 DS0 channels together. A further improvement was made with Extended Superframe (ESF) which groups all 24 channels of the T1 together.
As mentioned in the previous section, framing and multiframing on E1 lines is somewhat different. Framing information is carried in the first channel (channel 0). Depending on the framing in use, additional signalling information may be carried in channel 16, and/or additional error checking may be included.
Table H.2 T1 Alarms Alarm Description LOS Loss of Signal. No pulses are detected on the line for some period of time (100-250 bits times). OOF Out of Frame. The framing bits are in error. Reset or restored when the framing is restored. LOF
Loss of Frame. An OOF condition has persisted for 2.5 seconds. Restored when framing is restored for some period of time, 1-15 seconds. AIS Alarm Indication Signal. A string of ones sent as a Keep Alive. The device sending the AIS is unable to find the expected framing on its incomming signal, or is otherwise unable to forward what is presented to it. (Perhaps the sender of the AIS is in loop-back mode on the incoming port, for example). In any of these cases, the device will send an AIS down the line in the direction in which it would have forwarded the incoming data, if the data had not been corrupt (out of frame) or absent (loop-back, loss of signal). The Keep Alive helps to keep the network synchronized. RAI
Remote Alarm Indication. The remote end is telling you your signal is not being recieved.
Link Layer of T1
In chronological order of development, the three most important link layer protocols on the T1 WAN are High-level Data Link Control protocol (HDLC), Frame Relay, and Point to Point Protocol (PPP). PPP is sometimes consdiered an extension of HDLC, and in fact, HDLC, in one form or another, is a part of nearly every WAN protocol.
The ISDN link layer protocol is specified in Q.921, with aspects of the network layer specified in Q.930. Call set-up signalling on the D Channel is covered in Q.931. Essentially, ISDN uses some number of DS0s as B channels (bearer channels), to carry the user data, and a single D channel (data channel) to carry call set-up, control, and related signalling.
X.25 was an early effort to create a public data network using the existing telephone network. It emphasizes reliable delivery at the expense of data throughput. Having the Link Layer handle acknowledgements and the retransmission of any packets that remain unack'ed is a drag on throughput in X.25. The X.25 modulo 128 variant permits more data to be in flight unack'ed, and so improves throughput somewhat.
All of these link layer protocols make use of HDLC, or contain implementations of HDLC. The next section looks at a basic HDLC frame, and at Cisco's widely used implementation of HDLC.
HDLC and Cisco HDLC
The HDLC frame begins and ends with identical flags (0x7E). To prevent confusion between a flag and other data within the frame, any data containing more than five consecutive one bits has a zero bit inserted after the fifth one bit. Any zero appearing in the frame after five one bits is stripped out at the receiving end.
The Address field in most implementations contains nothing more than a statement of which end of a point-to-point link sent the frame. Some implementations may use this field for other things (addressing to a particular station in a multi-drop environment, for example), and the Address field can be extended to two bytes. When the address field is actually used to distinguish among possible recipients, stations ignore frames which do not contain their address.
Figure H.2 HDLC frame structure, in Asynchronous Balanced Mode (ABM)
The Control field is normally one byte, and describes the type of frame: Information (I), Supervisory (S), or Unnumbered (U). Information frames carry higher protocols such as TCP/IP. Supervisory frames handle flow control. Unnumbered frames are used to set up and tear down links and for miscellaneous functions.
The 3-bit N(R) element in the Control field for I-Frames and S-Frames is the sequence number of the received frame. The N(S) element in the Control field of I-Frames is the sequence number of the sent frame. When the Control field is extended to two bytes, the additional space is either used to extend the possible length of the sequence numbers, or it is reserved. These elements support reliable data transmissions in HDLC under ABM (Asynchronous Balanced Mode). ABM is "asynchronous" because nodes do not have to wait until a scheduled moment in order to communicate, and "balanced" because either end may initiate a conversation.
The P/F bit in the Control field is the Poll/Final bit. This is a legacy of the mainframe computing environment in which HDLC's predecessors evolved. Primary stations use this bit to force a response from secondary stations, and secondary stations use it to indicate that they are finished transmitting to the primary station.
The Code elements are used to send messages about the state of the transmission. Examples include: Receive Ready (RR), Receive Not Ready (RNR), Reject (REJ) and Selective Reject (SREJ). In combination with an included sequence number, these commands allow a modest degree of flow control.
A two-byte FCS-16 frame check sequence value appears immediately before the final flag byte.
HDLC formed the basis of many subsequent protocols, and is implemented in a variety of forms in many other protocols, including PPP, X.25, and Frame Relay. It heavily influenced the Link Access Procedures such as LAPB (...Balanced), LAPD (...on the D Channel, for ISDN), LAPF (...Frame), and LAPM (... Modem). HDLC also provided a model for the IEEE 802.2 LLC (Link Layer Control) protocol.
The original HDLC offered no clue as to the higher layer protocol it might be carrying. The address feature was meaningless on a link with only two ends, and the control features were redundant with those of higher layer protocols such as TCP. Cisco's implementation of HDLC addresses all of these issues and greatly simplifies HDLC.
In Cisco HDLC (cHDLC), the Address field is always one byte and takes only one of two values: 0x0F (unicast), or 0x8F (broadcast). This refers only to the encapsulated protocol data, as cHDLC does not support multi-drop. The Control field is one byte, set to zero. The Cisco implementation does not support HDLC windowing. A new two-byte Type Code field is added after the Control field to support the Ethertype code, naming the encapsulated protocol. The rest of the frame-flags and FCS-remains the same.
Figure H.3 Cisco HDLC frame
The following are some examples of how to interpret the LEDs on the T1/E1 Pod to troubleshoot potential problems.
LEDs on T1/E1 Pod
- The Signal light on the front panel is not illuminated. This could have several causes:
- If no lights are illuminated, check that the power cord is plugged into the electrical outlet.
- Check that the correct cables are used and that they are properly connected.
- Check that the router (with the internal CSU/DSU) is properly configured to the correct channels.
- The Signal light on the front panel is illuminated, but the Framing light is not (when using framed T1/E1 data links).
- Check that the router is in agreement with the carrier service agreement (line provisioning) and that the proper channels are configured.
- The Alarm light is illuminated (RED):
- If this is the Alarm light on the CPE side, check the timing configuration of the router.
- If this is the Alarm light on the Network side, check with the service provider for this network--something may be wrong with the T1/E1 line.
Supported protocols and physical interfaces
Table H.3 Supported protocols and physical interfaces Supported and tested Supported but not tested Protocols and Decodes Frame Relay PPP HDLC X.25 ISDN BRI and PRI Q.921 / Q.931 Physical Interface (DSU/router) V.35 X.21 (V.11, RS.422) 15-pin D connector V.24 (RS.232C) 25-pin D connector RS.449 37-pin D connector RS.530 25-pin D connector V.35 34-pin Winchester connector Physical Interface (from Telcos) T1 Fractional T1 E1 E1/G703 Fractional E1 ISDN PRI and BRI
T1/E1 Pod RJ-48 pin connections
Table H.4 T1/E1 Pod RJ-48 pin connections RJ 48 Connector CPE jack Network jack 1 Tx Rx 2 Tx Rx 3 Rx Tx 4 Rx Tx
Glossary of terms
Table H.5 Glossary of terms Term Definition CPE Customer Premises Equipment CSU/DSU Channel Service Unit/Data Service Unit. Typically packaged together as a single device connected to a digital line. The DSU performs protective and diagnostic functions for a telecommunications line and is required for both ends of a T1 or T3 connection. The CSU is connected to the WAN, the DSU to the customer equipment. DCE Data Circuit-terminating Equipment. The edge of the T1 or E1 WAN. From the perspective of a LAN administrator, it is the beginning of the WAN. From the perspective of the WAN administrator, however, it is where his circuit terminates. Traffic within a customer location moving in the "to DCE" direction is unambiguously understood to be headed onto the WAN. See also DTE DLCI Data Link Connection Identifier. In Frame Relay networks, a number that identifies one participating connection in the network. DLCI values are most often re-used, and are only temporarily and/or locally unique. By setting longer DLCI values, some Frame Relay networks make DLCIs global and permanent within their network. DS0 The basic unit of telecommunications link bandwidth, and equivalent to one telephone line. At 64,000 bits per second, a DS0 connection is the minimum required for clear, full duplex voice communications. DTE Data Terminal Equipment. The local equipment at a customer location such as a router or, in earlier days, a dumb terminal. Traffic moving in the "to DTE" direction is unambiguously understood to be headed off the WAN and into a local customer site. See also DCE Frame Relay High-performance WAN protocol that operates at the physical and data link layers of the OSI reference model. Frames are forwarded through a switch fabric with minimal flow control and error checking (but not correction), and no guarantee of delivery. Higher layer protocols handle error correction, retransmission, and so on. Frame Relay and LMI Frame Relay with Local Management Interface (LMI). LMI adds network management functions to earlier Frame Relay standards. In particular, LMI handles link integrity checks, as well as reporting and status check on the Permanent Virtual Circuit (PVC). HDLC High-level Data Link Control protocol. Based on the earlier IBM SDLC protocol, HDLC was implemented in a great number of variants as a simple and robust Link Layer protocol. The most popular current variant is probably Cisco HDLC (cHDLC). HDLC formed the foundation for PPP, which greatly extended the capabilities of this older protocol. Frame Relay and X.25 also make use of HDLC. IPARS (P1024B) International Program Airline Reservation System (IPARS), implemented as P1024B, a SITA implementation of Airline Line Control (ALC), the IBM airlines-specific protocol. P1024B uses 6-bit padded characters and IA/ TA (Interchange Address/ Terminal Address) for physical addressing. In this scheme, TA identifies the terminal and IA identifies the Agent Set Control Unit (ASCU), a cluster at the user side. ISDN Integrated Services Digital Network. A set of protocols which together define a digital connection over a variety of physical media. One advantage of ISDN is the ease with which bandwidth can be added to a service agreement in small increments. ITU-T International Telecommunication Union-Telecommunication Standardization Sector. PDH Pleisiochronos Digital Hierarchy. The totality of all the various, but closely related schemes for bundling multiple DS0s into larger units. For example, T1, T2, T3, E1, E2, and E3 are all part of the PDH. PPP Point to Point Protocol (PPP). Essentially incorporating HDLC, but with significantly expanded capabilities, PPP is used in various forms over a wide variety of serial connections. Specified and extended in Internet Engineering Task Force (IETF) Request for Comments (RFC) 1661, RFC 1662, and many others. PPP and LCP Point to Point Protocol (PPP) with its associated Link Control Protocol (LCP). Used in various forms over a wide variety of serial connections. Specified and extended in Internet Engineering Task Force (IETF) Request for Comments (RFC) 1661, RFC 1662, and many others. PVC Private Virtual Circuit. A virtual circuit that is "nailed up." That is, the virtual circuit is set up and remains established whether or not it is actively in use. Q.921/Q.931 Integrated Services Digital Network (ISDN) Link Access Procedure, D-Channel (LAPD). The name in the drop-down list refers to the Q.921 standard specifying LAPD, the signalling channel itself, and the Q.930 and Q.931 standards specifying the messages sent over the D channel. In ISDN the D Channel is used to set up and control ISDN calls. SITA SITA was originally known as "la Société Internationale de Télécommunications Aéronautiques," but now uses only its acronym. SNA Systems Network Architecture (SNA), an IBM protocol initially designed for mainframe and terminal computing. It preceded and heavily influenced both the OSI 7 layer model and the IEEE 802 family of LAN protocols. Traditional SNA specifies no physical layer, and is commonly run over PPP on T1 networks (for example, SNA Control Protocol (SNACP)). SVC Switched Virtual Circuit. A virtual circuit created on demand, like a telephone call. The circuit is set up and torn down each time, as needed. T1/E1 Pod Passive tap for T1/E1 links capable of extracting data from framed and unframed T1 or E1 circuits.. UTS (P1024C) Universal Terminal System (UTS) implemented as P1024C, the SITA implementation of the UNISYS UTS terminal protocol. It uses 7-bit (ASCII) characters and RID / SID (Remote Identifier / Station Identifier) for physical addressing. In this scheme SID identifies the terminal and RID identifies the Agent Set Control Unit (ASCU), a cluster at the user side. Virtual circuit A full duplex connection between two nodes in a switched network which is established without creating a continuous electrical circuit between the two ends (which would make it an actual circuit, rather than a "virtual" one). WAN Analyzer Card A PCMCIA card enabling WANPeek NX to provide WAN data capture and analysis. X.25 X.25 modulo 8. A 1976 CCITT (now ITU-T) specification for a public data network. Achieves high reliability by extensive buffering and retransmissions at a low layer, which in turn produces relatively high latency. Still widely used in Point of Sale communications. X.25 (mod 128) A variant of X.25, permitting a greater amount of data to be on the wire unack'ed at any one time, thus decreasing latency for larger data streams. Also an ITU-T specification.
WAN Addresses and Names
WANPeek NX recognizes three types of addresses: physical addresses, logical addresses, and symbolic names assigned to either of these.
The concept of an "address" on a Wide Area Networks (WAN) is quite different from that on a LAN or the Internet. Designed to work within the circuit-switched, point-to-point model of the telephone network, many WAN protocols (such as HDLC and PPP) have an extremely limited address function. Depending on the implementation, this may amount to little more than distinguishing between two ends of a circuit. Even when a larger address space is implemented, it is more in support of a multi-drop (primary to multiple secondaries) local environment, and has little to do with uniquely identifying a particular node in the vastness of global telecommunications.
The first major effort to create a uniform standard for packet switching (as opposed to circuit switching) over the existing telephone network was X.25. Even X.25 essentially sets up a telephone call to create a virtual circuit to its destination. ISDN also sets up its own calls, and offers some point-to-multi-point capabilities.
Frame Relay comes closer to the familiar model of the Internet, in that frames are routed through the switch fabric over a variety of possible paths to their destination. Frame Relay itself does not specify that destination as a fixed address, however. Higher level protocols must do that.
Frame Relay frames do contain a value called the Data Link Connection Identifier (DLCI). This identifies a connection to a neighboring device. In most Frame Relay networks, a DLCI value has only local significance and may be reused at other places in the network. DLCIs may be of different lengths, depending on the particular implementation of Frame Relay in use on a particular network. In the most common implementations, there are about 1,000 available DLCI values (apart from those reserved for signalling, for future use, and so forth). A single customer may be assigned multiple DLCIs, allowing them to use a single line to establish multiple simultaneous connections to the Frame Relay switch fabric or "frame cloud."
WANPeek NX treats DLCI values as physical addresses, but any analogy with an Ethernet MAC address is bound to be at least slightly misleading. DLCI identifies a connection, and only one DLCI value appears in a Frame Relay frame. Every frame on that particular connection is identified by that DLCI. By knowing which end of the conversation was assigned the DLCI and adding direction information, it is possible to see that a particular frame is being sent either from that DLCI or to that DLCI, but there is no need for either end of the conversation to mention any other DLCI value.
The Packets view of the Capture window in Figure I.1 shows DLCI values in the Source and Destination address columns.
Figure I.1 DLCI values displayed in a Capture window
A logical address is a network-layer address that is interpreted by a protocol handler. Logical addresses are used by networking software to allow packets to be independent of the physical connection of the network, that is, to work with different network topologies and types of media. Each type of protocol has a different kind of logical address, for example:
- an IP address (IPv4) consists of four decimal numbers separated by period (.) characters, for example:
- an AppleTalk address consists of two decimal numbers separated by a period (.), for example:
Depending on the type of protocol in a packet (such as IP or AppleTalk), a packet may also specify source and destination logical address information.
For example, in sending a packet to a different network, the higher-level, logical destination address might be for the computer on that network to which you are sending the packet, while the lower-level, physical address might be the physical address of an inter-network device, like a router, that connects the two networks and is responsible for forwarding the packet to the ultimate destination.
The following figure shows captured packets identified by logical addresses under two protocols: AppleTalk (two decimal numbers, separated by a period) and IP (four decimal numbers from 0 to 255 separated by a period). It also shows symbolic names substituted for IP addresses (www0.wildpackets.com and ftp4.wildpackets.com) and for an AppleTalk address (Caxton).
Figure I.2 Logical AppleTalk and IP addresses and symbolic names
The strings of numbers typically used to designate physical and logical addresses are perfect for machines, but awkward for human beings to remember and use. Symbolic names stand in for either physical or logical addresses. The domain names of the Internet are an example of symbolic names. The relationship between the symbolic names and the logical addresses to which they refer is handled by DNS (Domain Name Services) in IP (Internet Protocol). WANPeek NX takes advantage of these services to allow you to resolve IP names and addresses either passively in the background or actively for any highlighted packets.
In addition, WANPeek NX allows you to identify devices by symbolic names of your own by creating a Name Table that associates the names you wish to use with their corresponding addresses.
To use symbolic names that are unique to your site, you must first create Name Table entries in WANPeek NX and then instruct WANPeek NX to use names instead of addresses when names are available.
Other classes of addresses
When one says "address," one typically thinks of a particular workstation or device on the network, but there are other types of addresses equally important in networking. To send information to everyone, you need a broadcast address. To send it to some but not all, a multicast address is useful. If machines are to converse with more than one partner at a time, the protocol needs to define some way of distinguishing among services or among specific conversations. Ports and Sockets are used for these functions. Each of these is discussed in more detail below.
Broadcast and multicast addresses
It is often useful to send the same information to more than one device, or even to all devices on a network or group of networks. To facilitate this, the hardware and the protocol stacks designed to run on the IEEE 802 family of networks can tell devices to listen, not only for packets addressed to that particular device, but also for packets whose destination is a reserved broadcast or multicast address.
Broadcast packets are processed by every device on the originating network segment and on any other network segment to which the packet can be forwarded. Because broadcast packets work in this way, most routers are set up to refuse to forward broadcast packets. Without that provision, networks could easily be flooded by careless broadcasting.
An alternative to broadcasting is multicasting. Each protocol or network standard reserves certain addresses as multicast addresses. Devices may then choose to listen in for traffic addressed to one or more of these multicast addresses. They capture and process only the packets addressed to the particular multicast address(es) for which they are listening. This permits the creation of elective groups of devices, even across network boundaries, without adding anything to the packet processing load of machines not interested in the multicasts. Internet routers, for example, use multicast addresses to exchange routing information.
Figure I.3 Broadcast packets are processed by all nodes on the network
Some protocol types have logical Broadcast addresses. When an address space is subnetted, the last (highest number) address is typically reserved for broadcasts. For example:
IP Broadcast Addresses typically uses 255 as the host portion of the address; for example:
AppleTalk Broadcast Addresses use 255 as the node portion of the address:
While conceptually very powerful, broadcast packets can be very expensive in terms of network resources. Every single node on the network must spend the time and memory to receive and process a broadcast packet, even if the packet has no meaning or value for that node.
Figure I.4 AppleTalk broadcast and multicast packets
Multicast Address. In IPv4, all of the Class D addresses have been reserved for multicasting purposes. That is, all the addresses between 126.96.36.199 and 188.8.131.52 are associated with some form of multicasting. Multicasting under AppleTalk is handled by an AppleTalk router which associates hardware multicast addresses with addresses in an AppleTalk Zone.
Ports and sockets
Network servers, and even workstations, need to be able to provide a variety of services to clients and peers on the network. To help manage these various functions, protocol designers created the idea of logical ports to which requests for particular services could be addressed.
Ports and sockets have slightly different meanings in some protocols. What is called a port in TCP/UDP is essentially the same as what is called a socket in IPX, for example. WANPeek NX treats the two as equivalent. ProtoSpecs uses port assignments and socket information to deduce the type of traffic contained in packets.