Dialup ISP Archaeology

Posted: Jul 7, 2023

I sometimes see questions on Hacker News or other forums that are asking how early dialup ISPs worked (or some specific aspect thereof,) so I thought I’d write up what these systems were like. I worked at such an ISP from 1996 - 2006 and we acquired several others during that time, so I got to see a few different approaches.

The Bare Minimum

Roughly before 1996, most of our users dialed into our service using an analog modem (usually 14.4kbps or 28.8kbps) and accessed the Internet via a shell account on our Linux server. So, they needed a computer of some sort, a modem, and so-called terminal emulation software. There were a bunch of these programs (Telix was a popular choice on Windows) but they all did basically the same things:

On the ISP side, you needed:

In many cases, and certainly in our case, in the beginning those three things were all attached to a single Linux server. So, we had a Linux server with a dedicated 56kbps “leased line” – that was our connection to the Internet and a couple of modems for customers to dial in to. Based on the companies we subsequently acquired, and other early ISP employees I’ve talked to over the years, I think this was – far and away – the most common approach. Some typical circa-1995 hardware was:

The very first thing to fail in our setup was the 56kbps leased line. It didn’t really “fail” so much as become “totally saturated to the point of being unusable” at around the 8 - 10 concurrent user mark. Like many things in ISP land, the options for scaling this up were not linear and the upgrade – to a 1.536mbps T1 connection – required a leap of faith on the part of the owners. Prices for these varied wildly at the time, but a cost of $1,500/month was pretty typical.

T1s were generally connected to an external router, so our Linux server now needs an Ethernet card to interface with the router. This was after the days of vampire taps, but before the days of inexpensive hubs and switches, so our Ethernet network is going to be 10Base2. This is Ethernet over coaxial cable with BNC connectors. The way these worked is on each card you had a t-connector. BNC cables ran from card-to-card, and on the two computers at either end you had to add terminating resistors. If you disconnected either of the resistors or needed to add a host to the network, the entire Ethernet was unavailable until the resistors were properly re-attached (10Base2 Ethernet interprets the absence of the terminating resistors as a never-ending collision.)

The most common router manufacturers at the time were Cisco Systems and Wellfleet. Ours was a Cisco 2501, which was a popular single-T1 router. Well, it wasn’t actually a “T1 router”. It had a high-speed synchronous serial port and an Ethernet connection. You added an external CSU/DSU, made by a company like Adtran, to terminate the T1 and interface with the router.

By this time, it became more and more common for people to want to have the ability to run Internet client software directly on their PCs. Windows 95 had an IPv4 stack and modem dealer built-in. Before that, you needed something like Trumpet Winsock or The Internet Adapter to act as a Winsock (local networking API) to SLIP/PPP bridge. This let users run web browsers like NCSA Mosaic or Netscape Navigator and email software like Eudora directly on their computers. On the ISP side, you needed either an “terminal server” that had a SLIP/PPP implementation or equivalent software to run on your Unix server. We started out running The Internet Adapter, then SLIRP, and finally PPP software on our Linux server, then gradually migrated to an “access concentrator”.

I’ve mentioned “terminal server” a few times without really describing what these were, but essentially, they were proprietary appliances made by companies like Cisco Systems and Livingston Networks that contained a bunch of serial ports. Originally, these were used to either connect text-only terminals that users would use to interact with a host computer (usually a mainframe or minicomputer) or as a tool administrators would use to remotely manage various computers and appliances that had serial consoles. As it turns out, you could also connect modems to these, and allow users to dialin and, from either start a SLIP/PPP session, telnet from the terminal server to some other host, or connect to another attached serial port.

At ISPs, access concentrators were more common. These were devices like terminal servers, but with built-in modems and usually the ability to terminate ISDN connections as well. A key advantage of these systems was their density. Whereas external modems took up a lot of shelf space, and required an individual analog telephone line, access concentrators even in the 90’s could achieve densities of over 100 simultaneous connections in 3 rack units (5.25 inches tall by 19 inches wide,) and supported digital T1 interfaces, which made the cabling and provisioning of new telephone circuits much easier. The largest available units at the time could terminate a DS3 (672 circuits) in around half a standard rack.

The first access concentrator we bought was an Ascend Pipeline 400. This was exclusively for our ISDN customers and could terminate a single ISDN PRI. This was a T1 that could support up to 23 simultaneous 64kbps ISDN calls (the 24th channel is used for ISDN signaling.) With this, we could offer 64kbps and 128kbps ISDN service, in addition to our analog modem services. At the time, this device cost us the princely sum of $9,000. We also had to add a RADIUS daemon on our Linux server to authenticate the ISDN callers and make sure that only the users who paid for 128kbps could have two active simultaneous channels.

On the analog side, we continued to scale up our Linux-attached modem system for quite a while – at the end, we had 48 external Supra and USR Robotics modems in a single large dialup pool. These were all connected to the 60mhz Pentium server running Linux with three Cyclades SM-16 serial breakout adapters.

While this worked, there were a couple of problems that led us to eventually move on to use access concentrators for the analog modem service, as well:

Scaling this up Over the years, we had to further scale our uplink bandwidth, modem bank, and our server pool. We also added new devices to offer new services.

Bandwidth Once the single T1 was no longer sufficient, we went to a bonded T1 pair. This provided 3mbps of uplink bandwidth, but it wasn’t really “bonded” – it was just two T1s from the same provider with routing set up to send some connections over the first T1 and others over the 2nd. At the scale of a even a small ISP, this is perfectly fine – while a single TCP/UDP data stream can’t utilize more than 1.5mbps, in practice, that didn’t end up being a problem.

After that, there were various fractional T3 (45mbps) offerings available, which is what we eventually upgraded to. After that, you were in SONET land:

Modem Bank We already talked about this, but the technical path to scale this up was to move your modem lines to dedicated access concentrators with high-density DSP-based modems and T1/T3 delivered lines from the telephone company. Many ISPs, instead of buying their own concentrators, purchased wholesale dialup service from a larger ISP. These companies would point to your RADIUS servers and either provide or let you bring in a vanity number, giving your users a unique telephone number to dial for service.

Servers We gradually added servers and moved services to dedicated or semi-dedicated servers. Our path looked something like this:

Other Approaches In addition to our shoestring budget approach, I saw a few other approaches over the years:

Name: This will be displayed with your post
Email: This isn't visible to or shared with anyone except me (the site owner)
Comment: