- 234comments
- 242comments
- 40comments
- 3comments
- 65comments
- 264comments
- 12comments
- 43comments
- 76comments
- 52comments
- 181comments
- 51comments
- 245comments
- 4comments
- 14comments
- 53comments
- 195comments
- 19comments
- 169comments
- 22comments
- 90comments
- 32comments
- 3comments
- 16comments
- 181comments
- 151comments
- —discuss
- 19comments
- —discuss
- 303comments
Once upon a time when I was really really green we had this little lab datacenter, just a room full of machines, and the lab manager guy, crazy dude that I was a bit afraid of, handed me his pager for the weekend, I guess he had to be gone. I didn't really get this whole on-call thing back then, and so I shoved it into backpack and forgot about it. Well, there was a windstorm, so Sunday evening I check it, and there is a sequence of 24 messages from the UPS "power lost, shutting down server X#", with a gap of a few hours, then 23 messages from UPS "power reappeared, booting up server X#". As I said, I didn't really get this whole on-call thing so I ignored it until Monday when I manually booted server X24. Nobody noticed. Glad it wasn't that important...
Reminds me of one time I was working as a teen, for a workstation software development group, and my time was shared as an assistant to the site sysadmin.
The head sysadmin was gone and there was an incident that required rebooting or power-cycling the critical Sun 4/390 server that was in a locked machine room, so people were asking me what to do, but I hadn't yet been allowed to touch that server.
This machine handled email, special Internet gateway, Usenet, Sun YP maps, license servers, most of the critical NFS volumes (home directories, tools, SCM of our code), and maybe some weird thing for the non-engineering people who only had Windows boxes.
At the time, I thought of power-cycling as risky (for one thing, we had enough SPARCstations with Quantum 105 drives, that a rare power outage meant that we'd usually lose a workstation, though the 4/390 used huge IPI drives). And I think I didn't have that root password, and maybe not even the door keypad code that the console was behind.
So I mostly just speed-walked around the building, as if with purpose, while trying to figure out who I could contact, to get access and tell me what the safe procedure was to reboot/cycle the big loud machine that everything depended on.
Don't leave us hanging! What happened next!?
On my very first admin job all the way back in high school the single server with all the schools data had hung. So I power-cycled it only for it to blow the UPS and the fuse. Probably too much dust had accumulated in the PSU and the current surge of booting shorted it out. The following conversation with the adult in charge was not pleasant. When you press the reset button and the lights above go out with a bang - a stressful day.
These days most anything that would serve a role as a core router at a mid to large sized ASN will be powered from a huge battery bank and -48VDC rectifier system. You can easily have 400-800Gbps of traffic going through one box.
HN hug-of-death?
Here's an archived copy for those who may be having difficulty accessing the site: https://archive.ph/Ou23w
IMP is apparently the "Interface Message Processor": https://en.wikipedia.org/wiki/Interface_Message_Processor
And if you look at the photo of the front panel, you can see the "AUTO RSTRT" switch that's mentioned in the article - it's the third from the left on the top panel:
https://upload.wikimedia.org/wikipedia/commons/0/04/Interfac...
We had one of these refrigerator-sized boxes in our machine room in the early 80's to connect our DEC-20 to the Arpanet. A few years later, it was replaced by a Cisco router that was a fraction of its size.
Fun point from that page:
On that note, it's actually quite interesting to peruse the first few RFCs (https://www.rfc-editor.org/rfc-index-100a.html). Internet RFCs pretty clearly started off as the "Network Working Group"'s internal mailing list, where they'd throw any old thing: plans (https://www.rfc-editor.org/rfc/rfc4.html), meeting notes (https://www.rfc-editor.org/rfc/rfc6.html), mirrors for internal use (https://www.rfc-editor.org/rfc/rfc12.html), storytelling (https://www.rfc-editor.org/rfc/rfc89.html), etc.
It was also pretty clear that the IMP itself, and BBN as its vendor, was kind of at the center of this whole thing, where RFCs were always implicitly for internal discussion between these people who were also always in external discussion with BBN, recording what BBN said and what changes to the IMP were coming that would allow for/enable changes to planned protocols and software.
You might say that the IETF was really just a user group for "amateur" BBN IMP enthusiasts. :) I say "amateur", because BBN's real paying customers, as a DARPA contractor, would have been government, military, and telcos. Thus why none of those were participants in the "Network Working Group" — they were getting white-glove service directly from BBN!
If anyone knows exactly when DARPA itself (and US gov/mil installations as a consequence) began participating in the IETF / switched their own internal networks to using IETF-developed standards, that'd likely be a fascinating little point of history to learn about.
When I worked at BBN in the late 90s, around the time they were bought by GTE, they had a warehouse of old equipment, and I noticed they had a couple of IMPs. I really hope those ended up in a museum.
Looks like we may see more and more HN submissions that are URLs pointing to Mastodon "toots". Better than URLs pointing to Twitter pages, but still baffling given that the content is less than 500 characters, and even less in the case of Twitter.
The text of a toot is in the page. There is no Javascript-triggered HTTP request to retrieve it. One does not need to enable Javascript to fetch and read toots in a web browser. But, like Twitter, some web developer(s) thought it would be a good idea to try to prevent www browser users from seeing any content unless they enable Javascript.
One workaround is to simply add /embed to the URL.
https://mastodon.laurenweinstein.org/@lauren/109588605178700...
Personally I have added a rule to the localhost-bound forward proxy that adds this automatically so Mastodon toots are readable in the text-only browser.
are we looking at the same page? the text loads behind a spinner. and it’s quite slow
I think that's the point -- the link with the trailing embed loads much faster, nearly instantly.
The content here is almost 2,000 characters. Mastodon is just software you run yourself, even if it has a default character limit you can configure it to be whatever you want. Using Mastodon as a blog is actually kind of neat, although for my own use I'd want to customize it to have syntax highlighting for code.
Adding /embed to the Mastodon toot also improves the desktop reading experience by removing the left and right sidebars.
(It could be further improved if it had mobile-specific styling.)
"The Internet sees shutdowns as damage, and routes around them."
I don’t understand this quote which I’ve heard 1Mx. I’m not aware of any layer or protocol in networks that have a concept of damage. There is such a thing as failover links but they’re very rare for most layers of the network and certainly not unique to the Internet, they definitely have nothing to do with censorship which is the version of the quote I see the most.
It's a misquote. The actual comment was made about USENET, not the Internet, in response to an incident at Stanford. This was the "rec.humor.funny" incident, in the late 1980s.[1] Someone at Stanford (Ralph Gorin, an IT manager?) had blocked some USENET groups from entering the campus USENET network. But there was a faculty member who had a low-bandwidth connection to the outside world, and ran a USENET node. USENET works by syncing; when two nodes talk, they exchange any missing messages in any groups they both support. USENET nodes often connected by dialing each other with dial-up modems every hour or so. Some low-bandwidth connection automatically copied across the missing censored messages. So, USENET really did treat censored messages as lost data to be transmitted over another path.
John Gilmore is generally credited with saying this about the Internet around 1993, but he acknowledges was originally made about USENET.[2]
If anyone can find Stanford SU-SCORE archives for the 1980s, the original source is probably in there.
[1] http://jmc.stanford.edu/general/rhf.html
[2] https://quoteinvestigator.com/2021/07/12/censor/
I should've put an emoticon. It was a play on the old saying about trying to censor on the Internet. In this case, it was trying to "shut down the Internet", and "the Internet" popped right back on.
The author, Lauren Weinstein, has been involved with online policy/rights/responsibility issues since Usenet days, before the Web.
I was pleasantly surprised to see them on the Fediverse, but I guess that shouldn't be a surprise.
I was pleasantly surprised to see a self-hosted instance of Mastodon, though perhaps less surprising given the background of the author.
In case you're wondering (as I was):
Node 1: UCLA (30 August, hooked up 2 September, 1969) - SDS SIGMA 7, SEX
Node 2: Stanford Research Institute (SRI) (1 October) - SDS940/Genie
Node 3: University of California Santa Barbara (UCSB) (1 November) - IBM 360/75, OS/MVT
Node 4: University of Utah (December) - DEC PDP-10, Tenex
https://www.zakon.org/robert/internet/timeline/
And a network sketch:
https://www.scientificamerican.com/gallery/early-sketch-of-a...
I just finished reading "Where Wizards Stay Up Late - The Origins of the Internet", and the story of the IMP features heavily in that book. And it describes some of the resiliency features too, including the auto restart, and even auto loading of it's software! Cool stuff. I highly recommend it to anyone who wants to hear more about this time period of the early "internet". Good stuff.
Back then, it was spelled "Internet". Frmr VP Al Gore described it as "the information superhighway" while the late Sen. Ted Stevens called it "a series of tubes."
The transition from BBSes (with UUCP and CompuServe) to dial-up ISPs was tedious. Newsgroups (NNTP), IRC chat, FTP servers, and Gopher search clients were as important as web pages.
And placeholder icons, red-on-yellow scrolling marquee Comic Sans, page visit counters, and blink tags abounded. (Oh, and page rings, broken deep links, and capture auto-redirect pages. That was before the era of pop-under popup pages, embedded plugins (SWF Flash, ActiveX, Java Applets (which could be cryptographically-signed) controls, and sensible browser security controls.)
Around 1996, I wrote a shiny-new HTTP/1.0 (!) CGI chat server in Perl with a HTML-JavaScript client using (inefficient) polling. The client exploited HTTP/1.0 to keep the server connection alive almost indefinitely with periodic no-op messages. The server could push new chat messages by writing them to all N clients.
Later on, on LANs, multicast UDP could sometimes deliver a message to multiple listeners.
Multicast, SCTP, UDP lite, and broadcast don't really scale across the series of tubes. UDP is "best effort" and not guaranteed. UDP's advantage is lower latency. IPv6 will take another 30 years to deploy unless a collective of telcos conspire to phase out IPv4.
Edit 3: I believe I worked with someone who worked with Lauren at UCLA.
Edit 4: The first Internet message from UCLA to SRI was "LO".
BBN, that is an acronym I haven't heard in years that brings back memories!
They've been around for a long time, but I didn't hear of them until the mid 1990's when I was running a system that ping'd all the major DNS servers (Internet Weather Report).
https://icannwiki.org/BBN
They wrote the first TCP/IP stack. Correctly. We got the Berkeley one instead with all its bugginess (which bugs we still work around) for dumb reasons.
I have not found out what the API would have been like, instead of socket() / connect() / listen() etc.
Lauren's outlook on social media is monochromatic, apocalyptic and incorrect in one respect.
When real names and pictures are required of everyone, cyberdisinhibitionism, the tendency for people to act horribly and without boundaries, is reduced and conversation becomes more civilized. Signalvnoise (Basecamp / DHH blog) points to this observation.
https://signalvnoise.com/posts/2205-there-is-an-inverse-rela...
Another thought piece:
https://ericsammons.com/internet-anonymity-the-good-the-bad-...
Interesting article about grandiose and vulnerable narcissism interacting with anonymity:
https://www.sciencedirect.com/science/article/abs/pii/S01918...
Anonymity is necessary for some discussions (like these), but it should not be pervasive or without boundaries.
https://lauren.vortex.com
Title should be "I turned off the internet".
This toot reminds me of the plot of Infocom's classic text adventure "The Lurking Horror":
Despite a terrible snowstorm, a young G.U.E. Tech student travels to the school's computer lab to work on his grad paper. However, something strange has happened. The file containing the student's document has been partially overwritten by the Department of Alchemy's files. At first the student's only goal is to retrieve his lost document, but soon he realizes that something far more sinister is occurring in the depths of the school building.
I posted this thread of spooky NSA networking stories, supporting links, and old emails a few years ago to the "30 spies dead after Iran cracked CIA comms network" discussion:
https://news.ycombinator.com/item?id=18376750
At the University of Maryland, our network access was through the NSA's "secret" MILNET IMP 57 at Fort Mead. It was pretty obvious that UMD got their network access via NSA, because mimsy.umd.edu had a similar "*.57" IP address as dockmaster, tycho and coins.
https://emaillab.jp/dns/hosts/
https://multicians.org/site-dockmaster.html
Whenever the network went down (which was often), we had to call up a machine room at Fort Mead and ask them to please press the reset button on the box labeled "IMP 57". Sometimes the helpful person who answered the phone had no idea which box I meant, so I had describe to him which box to reset over the phone. ("Nope, that didn't work. Try the other one!" ;) They were even generous enough to issue us (CS department systems staff and undergrad students) our own MILNET TACACS card.
On mimsy, you could get a list of NSA employees by typing "grep contact /etc/passwd", because each of their courtesy accounts had "network contact" in the gecos field.
Before they rolled out TACACS cards, anyone could dial up an IMP and log in without a password, and connect to any host they wanted to, without even having to murder anyone like on TV:
https://www.youtube.com/watch?v=hVth6T3gMa0
https://news.ycombinator.com/item?id=18376974
I found this handy how-to tutorial guide for "Talking to the Milnet NOC" and resetting the LH/DH, which was useful for guiding the NSA employee on the other end of the phone through fixing their end of the problem. What it doesn't mention is that the key box with the chase key was extremely easy to pick with a paperclip.
Who would answer the Milnet NOC's 24-hour phone was hit or miss: Some were more helpful and knowledgeable than others, others were quite uptight.
Once I told the guy who answered, "Hi, this is the University of Maryland. Our connection to the NSA IMP seems to be down." He barked back: "You can't say that on the telephone! Are you calling on a blue phone?" (I can't remember the exact color, except that it wasn't red: that I would have remembered). I said, "You can't say NSA??! This is a green phone, but there's a black phone in the other room that I could call you back on, but then I couldn't see the hardware." And he said "No, I mean a voice secure line!" I replied, "You do know that this is a university, don't you? We only have black and green phones."
https://news.ycombinator.com/item?id=18376916
I dug up an "explosive bolts" reference -- fortunately that brilliant plan didn't get far.
Milo Medin knows this stuff first hand:
https://web.archive.org/web/20180505024303/https://innovatio...
https://news.ycombinator.com/item?id=18376885
There were rumored to be "explosive bolts" on the ARPA/MILNET gateways (whether they were metaphorical or not, I don't know). Here's something interesting that Milo Medin wrote about dual homed sites like NSA and NASA, that were on both the ARPANET and MILNET:
Here's another funny story from my email archives of around the same time, about how Jordan Hubbard's infamous rwall almost got UC Berkeley cut off from the internet, with some more interesting details from other old net boys like Milo Medin, Marc Crispin, and Dennis G. Perry:
https://news.ycombinator.com/item?id=31822138
Speaking of YP (which I always thought sounded like a brand of moist baby poop towelettes), BSD, wildcard groups, SunRPC, and Sun's ingenuous networking and security and remote procedure call infrastructure, who remembers Jordan Hubbard's infamous rwall incident on March 31, 1987?
https://news.ycombinator.com/item?id=25156006
https://en.wikipedia.org/wiki/Jordan_Hubbard#rwall_incident
I was logged in on my Sun workstation "tumtum" when it happened, so I received his rwall too, and immediately sent him a humorous email with the subject of "flame flame flame" which I've lost in the intervening 35 years, but I still have a copy of his quick reply:
Here's the explanation he sent to hackers_guild, and some replies from old net boys like Milo Medin (who said the program manager of the Arpanet in the Information Science and Technology Office of DARPA Dennis G. Perry said they would kick UCB off the Arpanet if it ever happened again), Mark Crispin (who presciently proposed cash rewards for discovering and disclosing security bugs), and Dennis G. Perry himself:
Also:
http://catless.ncl.ac.uk/Risks/4.73.html#subj10.1
https://everything2.com/title/Jordan+K.+Hubbard