you could maybe show some testing? if that's true, then it should be measurable

Test yourself, they link to their site from their profile and it's definitely delayed.

https://twitter.com/nytimes

A good test might include a bunch of domains. And checking the timing on each. Could we demonstrate the delay is on t.co and not on NYT?

I went through my own Twitter feed, and found 10 non-NYT links. All redirected almost immediately through t.co via wget, only lagging on the destination sites.

I also tried 5 NYT links. All had a very consistent 5 second delay through wget.

I could do more, but I don't care to. Everyone knows Elon has gone redpill, so it wouldn't surprise me if he's "owning the libs", but there also could be a dozen other reasons Twitter might do something like this (including plenty that are not nefarious). I just don't care to dig more...

Edit: I suppose I could have given the specific URLs, but I don't know if/how much t.co links leak info, so I'm not keen to do that. But the delay is absolutely on t.co and not the destination sites, at least as far as external users are concerned. It's possible that t.co queries the sites first before redirecting, and if e.g. the NYT is throttling their traffic that's what's delaying things. I don't know how to disambiguate that, but it's definitely a theory worth considering...

Actually, re: my edit - I think it really is worth considering whether there might be an accidental delay here that's on the NYT side. It's totally possible that Twitter is hitting the sites it redirects people to before it actually sends them there, for either analytics purposes or otherwise, and I'd trust the NYT devops less than Twitter's w.r.t. making sure things were fast.

Incompetence before malice, etc...

Just tried this - loaded almost instantly. But that's only the link in the profile. Not shared links in tweets.

Leaning towards there's something else going on deep in the DNS/ad servers/cdn/who knows. Not the first I've seen/heard of resolving delays with t.co... maybe it's even just something with legacy non-SSL links being redirected etc

You’re really twisting yourself into a buzzword cloud there aren’t you?

The link you clicked in the NYT bio is not a t.co link - I assume you noticed that but still are using it as counter-proof?

Remember when people were excoriating Google AMP for encouraging walled gardens? If true, this seems in so much worse faith than that.

Not worse. They are both as evil as it gets. Typical: take public resource and use it for an exclusive profit.

What happened to net neutrality? Could it applied for this case?

i mean, you can stop visiting the site, no? just leave, bro. it's not that hard. there are other means to connect to people.

You can both stop using the site as a user and still call out the awful stuff they do.

You're right! Which is why making Twitter's product worse when there are active competitors taking big bites out of their business seems... dumb?

Unfortunately not. All of my local government agencies - Police, Fire, DOT, Weather Service, Emergency Management updates, etc. are exclusively on twitter - they frequently post things there they don't even post to their own websites.

The speed at which enshitification is being unleashed surprises me each and every day.

Enshitification is different. It’s when companies destroy a product with hundreds of changes that prioritise internal politics above what end users want.

This is something else - just the ego of one rich guy petulantly satisfying his inner demons.

"Porque no los dos?"

A five-second delay may be enough to cause a measurably increase in the "stickiness" of Twitter if some people wait <5 seconds before clicking or scrolling onwards to something else.

Then they spend more time generating ad-revenue for Twitter than if they had gone off to the New York Times or something and started browsing over there.

Another great reason to get off Xitter.

can confirm from europe. shouldn’t this be illegal in america?

Net neutrality was repealed in 2018.

And then reinstated again at the state level. It never went away, this just doesn't apply because Twitter isn't an ISP (in that sense).

TIL! Looks like its only some of the states, and that is a good point about Twitter not being an ISP. Twitter is incorporated in Delaware which has not passed state-level net neutrality, so I don't know if it would have mattered anyway.

I can confirm. NYT shows a five-second redirect delay: "wget https://t.co/4fs609qwWt". It redirects to gov.uk immediately: "wget https://t.co/iigzas6QBx"

I almost didn't believe OP, because it's so comically inept and petty. But, I can also confirm in some private testing there is a deliberate delay.

Considering how common it is to deliberately break (not just with a "you log in now" modal, but not loading content, spinning forever, breaking layouts, etc) websites for non-logged-in mobile users, that's exactly how petty in imagine them to be. Twitter and Reddit do it, and Imgur comes and goes so I can't decide if for them it's deliberate or just incompetence.

Im not getting the same time delay with curl

- `time wget https://t.co/4fs609qwWt` -> `0m5.389s`

- `time curl -L https://t.co/4fs609qwWt` -> `0m1.158s`

And now add browser user-agent to the curl request and watch how slow it gets.

- `time curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -L https://t.co/4fs609qwW` -> 4.730 total

- `time curl -L https://t.co/4fs609qwWt` -> 1.313 total

Same request, the only difference is user-agent.

your URLs are different.

It’s about 4.5 seconds for me

https://imgur.com/a/qege0O9

Any DNS resolver libraries have a 4.5 second timeout? Maybe their infrastructure is just rotting.

It'd be weird that it's limited to specific outbound domains in that case.

domains load balanced across different servers in some overly complicated distributed topology with only one of them busted?

although seems unlikely it just happens to be the NYT.

glibc defaults to 5 sec, but the server wouldn't need to resolve the redirect domain -- that'd be the client's job. Unless it's doing so for some proprietary reason, of course.

Or did you mean failing to resolve some internal service's hostname?

Oddly enough the delay is reduced to 1 second by using curl's useragent string (wget --user-agent='curl/8.2.1' https://t.co/4fs609qwWt)

Seeing this makes me wonder if it's some sort of server-side header bidding ad server gone haywire, rather than something nefarious. Why would they only delay browser agents otherwise?

Probably a phishing/malware scan gone wrong then. NYTimes has Twitterbot in its robots.txt which might be related?

Even if it's deliberate, I don't see how people can complain. Google has outright blocked Breitbart for years. They prevent results from that domain from appearing at all unless you specifically force it with site: and apparently HN does the same. Politically motivated censorship and restricting "reach" is just how Silicon Valley rolls. Pre-Musk Twitter did freeze the New York Post's account and many other much worse things. It'd be a shame for Musk to be doing this deliberately, even though it seems unlikely. But that's the problem with creating a culture where that sort of behavior is tolerated, isn't it? One day it might be turned around on you.

Does the value Breitbart adds to the internet outweigh the negatives of turning people into dangerous fascists by weaponizing misinformation? No.

Does the value added by sources like the NYT outweigh the negatives of being occasionally biased or outright wrong? Yes.

NYT has arguably done far more damage to the world (Iraq, etc.) than Breitbart has.

There is no objective, public, or shared "value" at play here.

The only "values" that matter are the personal whims of whoever happens to own Twitter, or Google or Facebook.

Just because this is a very difficult question doesn't mean we can throw our hands up and pretend it doesn't exist. Many things in life are very difficult and yet worth solving anyway.

I didn't say the concept cannot exist: I said it's not at play here.

What gets a website censored, in the modern corporation-dominated Internet, is going against the interests and preferences of Big Tech owners - and nothing else. Nobody with any power is bound to look out for the public interest, however defined; ICANN is perhaps the only exception that comes to mind.

We can waste our time and attention debating over which targets were more or less deserving of censorship, based on our personal ideas of public interest. But as long as Big Tech is allowed to exist in its current form, we're like powerless peasants arguing about the decisions of kings.

Does the value the NYT adds to the internet outweigh the negatives of turning people into dangerous communists by weaponizing misinformation? No. Does the value added by sources like the Breitbart outweigh the negatives of being occasionally biased or outright wrong? Yes.

Could it just be rotting infrastructure? I.e there is some logic on most visited domains to allow ease of moderation, that logic is read heavy and is now bucking under skew.

Or even like some junior dev removed an index

A few years ago I remember their URL shortener on android app directing somewhere that my hostfile adblocker would catch (like an analytics domain or something). This made it so first click on certain twitter links would fail, but if I clicked it again it would go successfully. Ultimately I never researched it deeply enough but my guess is they had some sort of handler that would log whether loading their analytics service failed and serve up the direct link on the second attempt.

Is there some cache going on? On my first attempt, there is a 5 second delay. When I try it second time immediately it works without the 5 second delay. But if I try again after an hour, 5 second delay again!

Now do additional testing by adding and setting the http referer to t.co or twitter. Is it Twitter, or is it NYtimes doing this?

I don't see it:

  % curl -gsSIw'foo %{time_total}\n' -- https://t.co/4fs609qwWt https://t.co/iigzas6QBx | grep '^\(HTTP/\)\|\(location: \)\|\(foo \)'
  HTTP/2 301 
  location: https://nyti.ms/453cLzc
  foo 0.119295
  HTTP/2 301 
  location: https://www.gov.uk/government/news/uk-acknowledges-acts-of-genocide- committed-by-daesh-against-yazidis
  foo 0.037376

I think Twitter, err, X, just turned off the delay now that it's getting big media attention. I could reproduce it over and over again a little earlier, but now I can't anymore: https://news.ycombinator.com/item?id=37138161

[Edit:] I'm still seeing it with threads.net:

  curl -v -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/DzIiCFp7Ti

I don't see it with your URL either:

  % curl -gsSIw'foo %{time_total}\n' https://t.co/DzIiCFp7Ti | grep '^\(HTTP/\)\|\(location: \)\|\(foo \)'
  HTTP/2 301 
  location: https://www.threads.net/@chaco_mmm_room
  foo 0.123137

Doesn't matter if I do a HTTP/2 HEAD or GET:

  % curl -gsSw'%{time_total}\n' https://t.co/DzIiCFp7Ti 
  0.121503

HTTP/1.1 also shows no delay:

  % curl -gsSw'%{time_total}\n' --http1.1 https://t.co/DzIiCFp7Ti
  0.120044

I chalk this up to rot at X/twitter that is being fixed now that it was noticed.

I don't see it with your URL either

That's because you're not spoofing the User-Agent to be a browser rather than curl.

Oh that's it, thanks! In fact it it returns a 200 not 301 then:

  % curl -gsSw'%{time_total}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/DzIiCFp7Ti
  <head><noscript><META http-equiv="refresh" content="0;URL=https://www.threads.net/@chaco_mmm_room"></noscript><title>https://www.threads.net/@chaco_mmm_room</title></head><script>window.opener = null; location.replace("https:\/\/www.threads.net\/@chaco_mmm_room")</script>4.690000
  % curl -gsSIw'%{time_total}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/DzIiCFp7Ti
  HTTP/2 200 
  ...
  content-length: 272
  ...
  x-response-time: 4524
  ...
  
  4.660211

The delay is not there for nyti.ms (anymore) but once you use the Safari UA it's handled as 200 response:

  % curl -gsSIw'foo %{time_total}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/4fs609qwWt https://t.co/iigzas6QBx | grep '^\(HTTP/\)\|\(location: \)\|\(foo \)'
  HTTP/2 200 
  foo 0.126043
  HTTP/2 200 
  foo 0.037255

It really does seem that twitter is adding a 4.5s delay to some sites from web browsers. Could be malicious, could be rot...

They both load instantly for me.

I think that HN itself also shadow flags submissions from a list of domains it doesn't like.

Try submitting a URL from the following domains, and it will be automatically flagged (but you can't see it's flagged unless you log out):

  - archive.is
  - watcher.guru
  - stacker.news
  - zerohedge.com
  - freebeacon.com
  - thefederalist.com
  - breitbart.com

Good.

Hacker News isn't an open-ended political site for people to post weird propaganda.

How's archive.is "weird propaganda"?

It isn't banned in comments - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que..., https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que..., etc.

We probably banned it for submissions because we want original sources at the top level.

And how was the decision made to ban Federalist, but not say Guardian or The Daily Beast? Do you have any process in place to ensure that your political biases don't influence the list, or you don't care about that?

We don't have "processes" at HN. The idea makes my skin crawl.

Plenty of both left- and right-wing sites are banned and/or downweighted on HN. When a site is primarily about political battle, we either ban it or downweight it. Which of the two we choose depends on how likely the site is to produce the occasional interesting article (in HN's sense of the word "interesting"). That's why The Federalist and World Workers Daily (or whatever it's called) are banned, while National Review and Jacobin are merely downweighted. Both the Guardian and Daily Beast are downweighted, btw, as are most major media sites.

If you or anyone thinks that HN moderation is unfairly ideologically biased, I'm open to the critique, but you guys need to first look at the site as it actually is, and not just look at your own pre-existing perceptions. Every data point becomes a Convincing Proof when you do the latter.

People think that when their team gets moderated, the mods are OMG obviously on the other side. The Other Side feels exactly the same way. This "they're against me" perception is the most reliable phenomenon I've observed on HN. Leftists feel it, rightists feel it, Go programmers feel it, even Rust programmers feel it. Literally the very-most-popular topic on HN at any moment is perceived by someone as Viciously Suppressed because of this perception. Stop and think about that—it's kind of amazing. Someone should write a PhD thesis.

If you or anyone thinks that HN moderation is unfairly ideologically biased, I'm open to the critique, but you guys need to first look at the site as it actually is, and not just look at your own pre-existing perceptions.

Since when has moderation actions and relevant data been made available to the lay public here? We cannot look at the site as it actually is. We either have to trust you or pound sand.

Stop and think about that—it's kind of amazing. Someone should write a PhD thesis about it.

Just because (you think) everyone feels persecuted doesn't mean you're doing a good job keeping things level. It's a common joke to make but it's just a joke. Similarly, if both a rampant nazi, and a fierce tankie hate you, that doesn't make you a bastion of democracy. "Fairness" doesn't mean pissing off everyone equally, and that is neither a necessary or sufficient condition.

These are just minor notes, don't take them too seriously

Would you be open to publishing the down-weighting values if they're based on politics not spam?

This is the murkiest part to me since it's not just a binary flag.

What do you mean, "even" Rust programmers feel it? Rust programmers feel it the most. This site is hopelessly biased against Go programmers and towards Rust programmers!

We probably banned it for submissions because we want original sources at the top level.

Then why web.archive.org isn't also banned? [1] And what about things which aren't available from the original source anymore?

[1]: https://news.ycombinator.com/item?id=37130420

Then why web.archive.org isn't also banned?

Because web.archive.org is generally used for...

... things which aren't available from the original source anymore.

While archive.is is generally used to bypass paywalls. These 2 websites have 2 very distinct missions and use-cases.

That's a good question. See https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... and dredmorbius's comment at https://news.ycombinator.com/item?id=37138346 re archive.org.

As for "why archive.org and not archive.is" - that's a bit of a borderline call, but gouggoug pointed out some of it at https://news.ycombinator.com/item?id=37130890. The set of articles which (a) are no longer on the web, (b) are not on archive.org, but (c) are on archive.is, isn't that big. Paywall workarounds are a different thing, because the original URLs are still on the web (albeit paywalled). For those, we want the original URL at the top level, because it's important for the domain to appear beside the title.

You're apparently in the middle of editing this as I'm replying, but I suspect I'm close to the mark here: <https://news.ycombinator.com/item?id=37138346>

Well, yes, many sites are banned on HN. Others are penalized (see e.g. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...). None of this is secret, though we don't publish the lists themselves.

Edit: about 67k sites are banned on HN. Here's a random selection of 10 of them:

  vodlockertv.com
  biggboss.org
  infoocode.com
  newyorkpersonalinjuryattorneyblog.com
  moringajuice.wordpress.com
  surrogacymumbai.com
  maximizedlivingdrlabrecque.com
  radio.com
  gossipcare.com
  tecteem.com

And you don't see that as censorship?

This forum never said they’re a free speech haven.

That's true, but it's a bit of an interesting question because "free speech" has different meanings. The thing to understand about HN is that we're trying to optimize for one thing: intellectual curiosity (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...). Given that, we're not "free speech" in the sense of "post anything about anything" - we have to moderate spam, flamewar, lame comments like "ok boomer", etc., because those detract from curious discussion.

On the other hand, no single political or ideological position has a monopoly on intellectual curiosity either—so by the same principle, HN can't be moderated for political or ideological position.

It's tricky because working this way conflicts with how everyone's mind works. When people see a politically charged post X that they don't like, or when they see a politically charged post Y that they do like, but which we've moderated, it's basically irresistible to jump to the conclusion "the mods are biased". This is because what we see in the first place is conditioned by our preferences - we're more likely to notice and to put weight on things we dislike (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...). People with opposite preferences notice opposite data points and therefore "see" opposite biases. It's the same mechanism either way.

In reality, we're just trying to solve an optimization problem: how can you operate a public internet forum to maximize intellectual curiosity? That's basically it. It's not so easy to solve though.

The word censorship has so many meanings that I have to ask what you mean by it before I can say whether I see it that way.

Is it censorship that the rules of chess say you can't poke someone's queen off the board? We're trying to play a particular game here.

The word censorship has so many meanings that I have to ask what you mean by it before I can say whether I see it that way.

Perhaps its one of those things that are hard to define. [1] But that doesn't mean clear cases don't exist.

Is it censorship that the rules of chess say you can't poke someone's queen off the board? We're trying to play a particular game here.

No, but it is clearly political censorship if you only apply the unwritten and secret "rules" of the game to a particular political faction. Also, banning entire domain names is definitely heavy-handed.

[1]: https://en.wikipedia.org/wiki/I_know_it_when_I_see_it

Censorship has a very defined meaning. To take the chess analogy, it would be like allowing one side to poke the queen off the board, and not allow the other. This is very much like what happens at HN today.

You're dang right, trying to play a particular [rigged] game here.

Would be nice if the lists were published though with a link to the list from the submission form.

The problem is that if you publish the lists it leads to more abuses. For example if spammers find out which sites are banned then they just post other ones.

For example if spammers find out which sites are banned then they just post other ones.

I don't think that makes sense. The supposed spammers can just try looking up whether their submissions show up or not when not logged in.

That also requires additional effort on the spammers’ part. Increasing cost of attacks is an effective defense strategy.

Increasing cost of attacks is effective against good faith people, not spammers.

Even Cory Doctorow made this case in "Como is Infosec" [1].

The only problem with Cory's argument is, he points people to the SC Principles [2]. The SCP contain exceptions for not notifying about "spam, phishing or malware." But anything can be considered spam, and transparency-with-exceptions has always been platforms' position. They've always argued they can secretly remove content when it amounts to "spam." Nobody has challenged them on that point. The reality is, platforms that use secretive moderation lend themselves to spammers.

[1] https://doctorow.medium.com/como-is-infosec-307f87004563

[2] https://santaclaraprinciples.org/

platforms that use secretive moderation lend themselves to spammers

how is that? i can understand it not being useful, but how would it help spammers?

In my experience, increasing cost or delay even a little bit cuts out a disproportionate amount of bad stuff.

I once had the domain 'moronsinahurry' registered, though not with this group in mind...

You're correct again. Spammers and bots are the most determined actors, so these secretive measures don't impact them.

In fact, such secrecy benefits spammers. Good-faith users never imagine that platforms would secretly action content. So when you look at overall trends, bots, spammers and trolls are winning while genuine users are being pushed aside.

I argued that secrecy benefits trolls in a blog post, but I don't want to spam links to my posts in the comments.

Most spammers aren’t that competent. Hiding their posts without telling them used to be very effective on Reddit (now Reddit tells them). I guess it’s the same on HN.

If that were right, then HN would be overrun by spam.

So you think secretive measures more often defeat spammers than trusting users? I'd argue HN's content could be a lot better than it currently is.

Content curation is necessary, but shadow moderation is not helping. When a forum removes visible consequences, it does not prepare its users to learn from their mistakes.

I'll admit, I find HN to be more transparently moderated than Reddit and Twitter, but let's not pretend people have stopped trying to game the system. The more secret the rules (and how they are applied), the more a system serves a handful of people who have learned the secret tricks.

Meanwhile, regular users who are not platform experts trust these systems to be transparent. Trustful users spend more time innovating elsewhere, and they are all disrupted by unexpected secretive tricks.

I think there are two different types of sites you are blocking: (1) those which are just pure spam; (2) news/opinion/etc websites that you’ve decided are not suitable for HN for various reasons (such as being low quality and tending to produce more ideological flame-wars than curiosity), for example Breitbart

I agree that publishing case (1) causes harm (spammers will just use a different domain if they know you’ve blocked theirs.) But case (2) is rather different. I don’t think the same justification for lack of transparency exists in this case. And I think shadow-banning the submission in case (2) is not very user-friendly. It would be better to just display an error, e.g. “submissions from this site are blocked because we do not believe it is suitable for HN” (or whatever). A new user might post stuff like (2) out of misunderstanding what the site is about rather than malevolence, so better to directly educate them than potentially leave them ignorant. Also, while Breitbart is rather obviously garbage, since we don’t know everything in category (2) on the list, maybe there are some sites on it whose suitability is more debatable or mixed, and its inappropriateness may be less obvious to someone than Breitbart’s (hopefully) is

[flagged]

Well, unless surrogacymumbai.com is where we post right-wing news from, that's hardly a problem, eh?

I posted https://news.ycombinator.com/item?id=37131212 before noticing that you'd already made the same argument - sorry! But I have to leave it up because penispowerworldwide.com makes me laugh.

That's what "random 10 out of 67k sites" gives you. Your set was cherry-picked, dang's wasn't.

That said, dailykos.com seems to be banned. Happy now?

Your set was cherry-picked

Not exactly cherry-picked, these were from things I submitted myself and noticed that were shadow flagged.

That said, dailykos.com seems to be banned. Happy now?

No, I'd be happy when archive.is, Federalist and the rest of the non-spammy ones are unbanned. (Also, even if "balanced" censorship was the desired goal, having a single unreliable left-wing source banned vs a ton of right-wing ones doesn't really achieve that.)

It is a secret if the system does not inform the poster it's been penalized.

What is it if the information is freely available, to anyone asking, for a single domain they are trying to post at that time?

It’s not secret, because they’ll be provided an answer if they email the mod team.

It’s not free as in open source, because it isn’t available for anyone to download and study in full.

So, since it’s not secret, is it public, or private? Since it’s not published in full but any query of LIMIT 1 is answered, is that open, closed, or other?

Restrictions to publication don’t necessarily equate to secrecy, but the best I’ve got is “available upon request”, which isn’t quite right either. Suggestions welcome.

Content moderation systems often hide mod actions from the content author [1]. That's a secret.

The opposite would be to show the author of the content some indicator that it's been removed, and I would call that transparent or disclosed moderation.

Interestingly, your comment first appeared to me as "* * *" with no author [2]. I wonder if that is some kind of ban.

[1] https://www.youtube.com/watch?v=8e6BIkKBZpg

[2] https://i.imgur.com/oGnXc6W.png

edit I know you commented again but it's got that "* * *" thing again:

https://news.ycombinator.com/item?id=37130675

https://archive.is/Eov7z

It's not a ban. It appears when the user has 'delay' in their profile set to N minutes and N minutes haven't elapsed yet. We should probably make this more explicit.

Re the 'delay' setting see https://news.ycombinator.com/newsfaq.html.

Suggestions welcome

"This domain is not allowed on HN" as an error message upon submission.

That’s not going to work, because now that’s an API for spammers to bulk process against a domain list. The only available API must be human communication to the mod team, or the spammers will overcome it with automation.

Spammers can already do that API call and see if the domain shows up. This only puts human users at the same level of consideration as spammer automation.

That's trivial to figure out.

It's quite possible the reason the list isn't public is because it would give away information about what thought is allowed and what thought isn't.

There's a lot of user hostile moderation practices that occur on this site, manual and automatic. They're not often, or really at all, discussed. Some of them don't work well, and haven't for as long as they've existed.

I don't want us to be user hostile. Can you link to some examples?

I would consider any moderation action that isn't visible to users to be user hostile.

If you're going to censor someone, you owe it to them to be honest about what you're doing to them.

You possibly haven't experienced how devious and determined and dishonest and unpleasant some bad actors are, including SPAMmers.

(Even when doing the RightThing(TM) would probably be easier...)

And, BTW, I occasionally get blocked by the mechanisms here, even though not doing anything bad, but understand that there is a trade-off.

I am fully aware of the issue.

That's one of the costs with having a public website.

If you're going to censor someone

unless HN is suddenly the government what you've misnomered is moderation, not censorship. Calling censorship just exaggerates your opinion and makes you look unhinged. It's a private website not national news.

Censorship is not limited to who does it.

HN operates, based on a number of reasons, on numerous dynamics of friction and nudges. Mostly for the better. I've had my disagreements about things in the past, though as I watch the site and have studied it (particularly over the past few months, see: <https://news.ycombinator.com/item?id=36843900>) I mostly agree with it.

The parts that don't work especially well, most particularly discussion of difficult-but-important topics (in my view) ... have also been acknowledged by its creator pg (Paul Graham) and mods (publicly, dang, though there are a few others).

In general: if you submit a story and it doesn't go well, drop a note to the moderators: hn@ycombinator.com. They typically reply within a few hours, perhaps a day or if things are busy or for complex.

You can verify that a submission did or didn't go through by checking on the link from an unauthenticated (logged-out) session.

if you submit a story and it doesn't go well, drop a note to the moderators

You can verify that a submission did or didn't go through by checking on the link from an unauthenticated (logged-out) session.

Trustful users do not think to do this, and it would not be necessary if the system did not keep the mod action secret.

Trustful souls may not.

Those who have been advised to do so, through the Guidelines, FAQ, comments, or moderator notes, do, to their advantage.

(I'd had a submission shadowbanned as it came from the notoriously flameworthy site LinkedIn a month or few back. I noticed this, emailed the mods, and got that post un-banned. Just to note that the process is in place, and does work.)

You don't see the harm of elbowing out trustful people from the public square?

What we do is try to educate them and loop them back in.

I've done this on multiple occasions, e.g.: <https://news.ycombinator.com/item?id=36191005>

As I commented above, HN operates through indirect and oblique means. Ultimately it is is a social site managed through culture. And the way that this culture is expressed and communicated is largely through various communications --- the site FAQ and guidelines, dang's very, very, very many moderation comments. Searching for his comments with "please" is a good way to find those, though you can simply browse his comment history:

- "please" by dang: <https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=tru...>

- dang's comment history: <https://news.ycombinator.com/threads?id=dang>

Yes, it means that people's feelings get hurt. I started off here (a dozen years ago) feeling somewhat the outsider. I've come to understand and appreciate the site. It's maintained both operation and quality for some sixteen years, which is an amazing run. If you go back through history, say, a decade ago, quality and topicality of both posts and discussions are remarkably stable: <https://news.ycombinator.com/front?day=2013-08-14>.

If you do have further concerns, raise them with dang via email: <hn@ycombinator.com> He does respond, he's quite patient, might take a day or two for a more complex issue, but it will happen.

And yes, it's slow, inefficient, and lossy. But, again as the site's history shows, it mostly just works, and changing that would be a glaring case of Chesterton's Fence: <https://hn.algolia.com/?q=chesterton%27s+fence>.

radio.com looks legit, what is wrong with it?

Why not publish the list? Users would know what not to submit in that case. Except maybe you’re worried about the list being heavily curated a certain way…

That's a lot of domains! Did you source that from some other list, or is that a result of 67k individual entries? Either way, I appreciate it.

Out of curiosity, what's the rationale for blocking archive.is? Legal reasons I assume?

That's a lot of domains!

Not really. 67k/350m=0.02%

And 0.02% is really a lot.

Is it?

0.02% of a big number ... is still a big number

0.02% of 10,000 is 2 - pretty small

0.02% of 1,000,000,000 is 200,000 ... kinda big :)

In my opinion, absolute value is not important here, what matters is the fraction. I consider 0.02% to be large by itself for the given context.

Maybe "major media" should include tech media like The Register, Ars Technica, Tech Dirt, etc.. Unlike with media like the NYT, Bloomberg or Reuters, I've never seen a story for which these sites were the best source and much of what they publish is blogspam summarizing stories that have already been posted on HN, usually with a votebait title.

Do you have get-out-of-jail or N-strikes-and-you're-out policies? What if someone's legitimate website gets caught in this? I've also long wondered about user specific shadow bans. Can you please shed light on this?

There's no automatic unban. That would require writing code that knows how to tell a good (for HN) site apart from a bad one, and if we could write such code, we wouldn't need to keep a list of banned sites in the first place. However, we're always happy to unban a site when we notice that it's actually fine for HN, or when someone points this out to us.

Re shadowbanning (i.e. banning a user without telling them), see the past explanations at https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... and let me know if you still have questions. The short version is that when an account has an established history, we tell them we're banning them and why. We only shadowban when it's a spammer or a new account that we have reason to guess is a serial abuser.

You forgot to mention that you are also shadowbanning the ability of users to upvote or downvote things when you dislike their upvotes or downvotes—instances that you perceive as not contributing to the discussion or that are escalating the conversation.

I didn't forget to mention that - it's simply not what the word shadowban means, as I've always understood and used it.

This is a big problem with trying to explain these things - people mean very different things by the same words, and it leads to misunderstanding.

Which other word do you think would be suitable here? In my view, 'shadowban' aligns with the definition in this context, as you aren't notifying people about it (hence 'shadow') and their actions of upvoting or downvoting have no impact (so same as shadowbanning comments or submissions etc).

I would call it either a penalty or a loss of voting privileges, depending on the specific case. It's not a ban because the account is not excluded from participating in other ways. In the same way, downweighted or penalized sites aren't the same as banned sites.

Thanks for the thoughtful reply. Is it also true that users with certain karma count or special permissions have more significant - and potentially lasting - downvoting weight that impacts to the downvoted party's long term reputation?

I'm interpreting your question as "are there privileged HN members with supervotes", excluding moderators, and who can single-handedly kill submissions or comments.

So far as I'm aware, no, and there are comments from dang and pg going back through the site history which argue strongly against distinguishing groups of profiles in any way.

The one possible exception is that YC founder's handles appear orange to one another at one point in time (pg discusses this in January 2013: <https://news.ycombinator.com/item?id=5025168>). The feature was disabled for performance reasons.

Dang mentions the feature still being active as of a year ago: <https://news.ycombinator.com/item?id=31727636>

I seem to recall a pg or dang discussion where showing this publicly created a social tension on the site, as in, one set of people distinguished from another.

dang discusses the (general lack of) secret superpowers here: <https://news.ycombinator.com/item?id=22767204>, which reiterates what's in the FAQ:

HN gives three features to YC: job ads (see above) and startup launches get placed on the front page, and YC founder names are displayed to other YC alumni in orange.

<https://news.ycombinator.com/newsfaq.html>

Top-100 karma lands you on the leaderboard: <https://news.ycombinator.com/leaders>. That's currently 41,815+ karma. There are also no special privileges here other than occasionally being contacted by someone. (I've had inquiries about dealing with the head-trip of being on the leaderboard, and a couple of requests to boost submissions, which I forward to the moderation team).

Thank you, dredmorbius, for this very helpful response. dang has mentioned a procedure or system that involves making guesses for the purpose of shadowbanning. I wonder if downvotes (edit: or post flags) from special users like the ones you mention are used as strong signals in that guess-making?

So, is there an algorithm to be features in the front page? —other than upvotes. If a site can be banned, can another one be promoted?

Understandable, but I think there should be some discriminating system for another class of sites, the "you can submit but not discuss" ones.

For example, a recent submission (of mine):

"Luis Buñuel: The Master of Film Surrealism"

it had no discussion space because (I guess) it comes from fairobserver.com . Now, I understand that fairobserver.com may had been an hive of dubious publishing historically, but it makes little sense we cannot discuss Buñuel...

Maybe a rough discriminator (function approximator, Bayesian etc.) could try and decide (based at least on the title) whether a submission from "weak editorial board" sites seems to be material to allow posts or not.

Oh I agree - https://news.ycombinator.com/item?id=36924205 was a fine submission. Can you please email hn@ycombinator.com so I can send you a repost invite for it?

That domain is a borderline case. Sometimes the leopard really changes its spots, i.e. a site goes from offtopic or spam to one that at least occasionally produces good-for-HN articles. In such cases we simply unban it. Other times, the general content is still so bad for HN that we have to rely on users to vouch for the occasional good submission, or to email us and get us to restore it. I can't quite tell where fairobserver.com is on this spectrum because the most recent submission (yours) is good, the previous one (from 7 months ago) is borderline, and before that it was definitely not good. But I've unbanned it now and moved it into the downweighted category, i.e. one notch less penalized.

pg posted an early version of the list back in March 2009 when it include only 2096 sites:

<https://news.ycombinator.com/item?id=498910>

That grew fairly rapidly, it was at 38,719 by 30 Dec 2012:

<https://news.ycombinator.com/item?id=4984095> (a random 50 are listed).

I suspect that overwhelmingly the list continues to reflect the characteristics of its early incarnations.

The difference is that HN is explicitly heavily moderated while Twitter pretends to be an equitable free speech platform.

Unless you disagree with Elon

Try submitting a URL from the following domains, and it will be automatically flagged (but you can't see its flagged unless you log out): - archive.is

I can assure you that is Not the case with HN: on posting archive.is URL's, proof?

Look at my comment postings : https://news.ycombinator.com/threads?id=archo

Is it possible you have been shadow-banned for poor compliance to the [1]Guidelines & [2]FAQ's?

[1] : https://news.ycombinator.com/newsguidelines.html

[2] : https://news.ycombinator.com/newsfaq.html

I can assure you that is Not the case with HN: on posting archive.is URL's, proof?

It's not banned in comments, but it is banned in submissions. @dang (HN's moderator) confirms that here: https://news.ycombinator.com/item?id=37130177

Isn't blocking Stacker.news a petty move?

It's basically HN, but you can earn small tips for submissions and comments.

It could be because they saw they were getting low quality links from there. In any case, since HN prefers original sources, it’s less likely that a news aggregator would be a good source (the occasional Reddit comment notwithstanding)

That sounds like a play on the "slow ban":

Selective downtime, where the troll finds that the website is down (or really slow) quite often. Not all of the time, because that would tip them off. Trolls are impatient by nature, so they eventually find a more reliable forum to troll.

https://ask.metafilter.com/117775/What-was-the-first-website...

Usually, slow bans aren't done for people who pay you money and add value to your platform.

NYT and others are still adding value to the platform even if users become discouraged from clicking those links and navigating away (choosing instead to just stay on Twitter and read the summary and reactions there). Nudges like this help Twitter have their cake and eat it, too.

Why not try a Nitter instance. It offers RSS and none of the URLs are shortened with "t.co".

They just broke nitter. Hopefully it will be fixed soon but in the long term Musk can cut nitter off if he wants.

It's working for me.

Which instance? None of the ones I'm trying are working. Maybe it's regional

You can refer to the full list on the github page[0] (which I find to be pretty accurate). I personally use La Contre-Voie[1] for no particular reason.

nitter.net was historically a little less reliable for me due to rate limiting, which is why I initially switched. They worked around the rate limiting issue now, so that may no longer be the case.

[0] https://github.com/zedeus/nitter/wiki/Instances

[1] https://nitter.lacontrevoie.fr/

Thanks. Unfortunately, not having any luck with any of them, including lacontrevoie. Nitter.net gives a nim error, the others don't find any tweets

If give us a Twitter username as an example I'll find an instance thats working. The tweets and the non-t.co URLs are in the RSS.

Thanks - try holly, for example.

I'd like to invoke Hanlon's razor here to at least cut down on the potential for over-the-top reactions that can be somewhat embarassing in retrospect:

"Never attribute to malice that which is adequately explained by stupidity."

Absolutely not applicable to corporations. Bad faith actions are rampant, and I am unwilling to give the benefit of the doubt to billion dollar companies.

How about incompetence instead of stupidity, would that apply to many billion dollar companies?

There are a number of reasons this could be that are not necessarily nefarious. It's odd to jump straight to "something evil is going on"

Tell me this, does Twitter have some kind of "play nice" code that slows down inbound clicks through to a site so it doesn't DDOS other sites? I can easily imagine a scenario where anti DDOS caode would allow small sites to pass through quickly, yet sites under heavy "click through" load are being slightly throttled.

This wouldn’t reduce total requests made so it would be a weird anti-DDOS measure.

Many words about site moderation have been posted in the thread.. Who here has actually moderate(s|d) a forum/chat site?

I have, for about 20 years. I empathise with @dang's methods and explanations here. e.g., don't disclose a blocklist, use shadowbanning, don't get dragged down by pedantic responses to explanations. You're often one person against many trolls finding grey areas for sport, or spammers (automated or otherwise) probing defences.

Worth pointing out that t.co has always been an instance of an annoying and seemingly unjustified practice I named "nonsemantic redirect". Rather than legitimately redirecting using an HTTP Location header, it instead is an HTML page with a META refresh tag on it.

You don't see this with curl/wget because they use user agent sniffing. If they don't think you're a browser they _will_ give you a Location header. To see it, capture a request in Firefox developer tools, right click on the request, copy as CURL. (May need to remove the Accept-Encoding tag and add -i to see the headers).

Could you explain what the intended/expected outcome is for this? What is accomplished by doing that?

The purpose is so that Twitter is seen as the source of the traffic. A lot of Twitter-sourced traffic comes from native apps, so when people click links from tweets, they usually don’t send referrer information.

If the redirects were server side (setting the Location header), a blank referrer remains blank. Client side redirects will set the referral value.

From Twitter’s POV, there’s value in more fully conveying how much traffic they send to sites, even if it minorly inconveniences users.

How does this inconvenience users? It sounds like you’re saying site owners will be able to distinguish between users with a blank referer and users whose “referer” was the desktop app. Ignoring the privacy angle, isn’t that a good thing?

Other than the privacy angle a meta redirect is always a bit slower than a location header. You need to send an html page (more bytes) that the browser needs to render and then act on (more work).

A location header is nearly unnoticeable, a meta refresh page gives you a flash of a blank interstitial screen.

(Not that I had the same annoyance, just explaining the difference to the end user of the two approaches)

No, in fact now t.co even returns an empty body with it's 301 response:

  % curl -vgsSw'< HTTP/size %{size_download}\n' https://t.co/DzIiCFp7Ti 2>&1 | grep '^< \(HTTP/\)\|\(location: \)'
  < HTTP/2 301 
  < location: https://www.threads.net/@chaco_mmm_room
  < HTTP/size 0

Firefox:

    <head><noscript><META http-equiv="refresh" content="0;URL=https://www.threads.net/@chaco_mmm_room"></noscript><title>https://www.threads.net/@chaco_mmm_room</title></head><script>window.opener = null; location.replace("https:\/\/www.threads.net\/@chaco_mmm_room")</script>

You didn't read the second paragraph of the comment you replied to-- which explained this exact issue before you replied "no":

You don't see this with curl/wget because they use user agent sniffing. If they don't think you're a browser they _will_ give you a Location header. To see it, capture a request in Firefox developer tools, right click on the request, copy as CURL.

People are still using Twitter?

The redirect has always been a bit annoying and I found on mobile if you click a link, see it load the t.co URL, then quickly go back and click again, it instead loads the actual linked URL. So I just instinctively do this now.

Additional details I wrangled for this rabbit hole. I don't think it's t.co doing this intentionally, but rather poor handling of 'do you have our cookies or not'. Everyone in this thread _proving things_ without taking into account the complexity of the modern web.

   man curl
       -b, --cookie <data|filename>
              (HTTP) Pass the data to the HTTP server in the Cookie header. It is supposedly the data previously received from the server in a "Set-Cookie:" line.

----

Add that option to your curl tests.

    ---
    $ time curl -s -b -A "curl/8.2.1" -e ";auto" -L https://t.co/4fs609qwWt -o /dev/null | sha256sum 
    eb9996199e81c3b966fa3d2e98e126516dfdd31f214410317f5bdcc3b241b6a2  -

    real    0m1.245s
    user    0m0.087s
    sys     0m0.034s
    ---

    $ time curl -s -b -e ";auto" -L https://t.co/4fs609qwWt -o /dev/null | sha256sum 
    eb9996199e81c3b966fa3d2e98e126516dfdd31f214410317f5bdcc3b241b6a2  -

    real    0m1.265s
    user    0m0.103s
    sys     0m0.023s
    ---

    $ time curl -s -b -A "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0" -e ";auto" -L https://t.co/4fs609qwWt -o /dev/null | sha256sum 
    eb9996199e81c3b966fa3d2e98e126516dfdd31f214410317f5bdcc3b241b6a2  -

    real    0m1.254s
    user    0m0.100s
    sys     0m0.018
    ---

Amen

I can replicate this behavior fairly easily in a browser.

  1. Open incognito window in Chrome
  2. Visit https://t.co/4fs609qwWt -> 5s delay
  3. Open a second tab in the same window -> no delay
  4. Close window, start a new incognito session
  5. Visit https://t.co/4fs609qwWt -> 5s delay returns

The reason there isn't a delay the second click is because the redirect is cached locally in your browser.

Your humble anonymous tipster would appreciate if you do a little legwork.

The reason there isn't a delay the second click is because the redirect is cached locally in your browser.

No, because it’s not an HTTP redirect. It’s an HTML page that redirects you using a meta tag, something that the browser doesn’t cache.

Your humble anonymous tipster notes to their skeptical audience that browsers are capable of caching all sorts of things, even something as peculiar as an HTML page.

browsers are capable of caching all sorts of things, even something as peculiar as an HTML page.

Yes, and this is irrelevant to your previous comment: caching the HTML doesn’t cache the redirect itself.

You can lead a horse to water, but you can't make him drink. The delay was not on the HTML page.

What is that attempting to prove or replicate?

Here's a simpler test I think replicates what I am indicating in GP comment, with regards to cookie handling:

Not passing a cookie to the next stage; pure GET request:

    $ time curl -s -A "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0" -e ";auto" -L https://t.co/4fs609qwWt > nocookie.html

    real    0m4.916s
    user    0m0.016s
    sys     0m0.018s

Using `-b` to pass the cookies _(same command as above, just adding `-b`)_

    $ time curl -s -b -A "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0" -e ";auto" -L https://t.co/4fs609qwWt > withcookie.html

    real    0m1.995s
    user    0m0.083s
    sys     0m0.026s

Look at the differences in the resulting files for 'with' and 'no' cookie. One redirect works in a timely manner. The other takes the ~4-5 seconds to redirect.

oh boy... -b takes an option which in your examples is -A and -e, then what follows is interpreted as a URL and you throw away the warnings:

  % curl -vgsSIw'> %{time_total}\n' -b -A "curl/8.2.1" https://t.co/DzIiCFp7Ti 2>&1 | grep '^\(* WARNING: \)\|\(Could not resolve host: \)\|>' 
  * WARNING: failed to open cookie file "-A"
  * Could not resolve host: curl
  curl: (6) Could not resolve host: curl
  * WARNING: failed to open cookie file "-A"
  > HEAD /DzIiCFp7Ti HTTP/2
  > Host: t.co
  > User-Agent: curl/8.1.2
  > Accept: */*
  > 
  > 0.013309
  > 0.112494

This is what I have come to expect from every person that calls themselves a "free speech absolutist." What they actually believe is that they should be able to say whatever they want and do whatever they want, personally, without any consequences for themselves. There is no grander principle than "my ability to do what I want and exert power over others however I want, without critique or criticism."

I really wish the term hadn't been polluted this way.

Disclaimer: I am not comparing Twitter to warlords, dictators or genocide. But this quote (from Lord of War) really encapsulates a lot of what you say:

Yuri Orlov: [Narrating] Every faction in Africa calls themselves by these noble names - Liberation this, Patriotic that, Democratic Republic of something-or-other... I guess they can't own up to what they usually are: the Federation of Worse Oppressors Than the Last Bunch of Oppressors. Often, the most barbaric atrocities occur when both combatants proclaim themselves Freedom Fighters.

Update: hours after being exposed and publicized in the Washington Post, the behavior has stopped:

On Tuesday afternoon, hours after this story was first published, X began reversing the throttling on some of the sites, dropping the delay times back to zero. It was unknown if all the throttled websites had normal service restored.

https://archive.is/2023.08.15-210250/https://www.washingtonp...

I call myself a free speech absolutist (or advocate at least, absolutist is more of a slur). False compromises belong in the past. What X is doing isn't free speech at all and they have stated that advertisers will dictate what content will be seen, there is no commitment to freedom of speech at all.

But at least I can hold them responsible for violating their own stated values. The former Twitter leadership just hid content that didn't fit theirs or third parties sensitivities and told me they are doing me a favor.

Restricting speech is always in the interests of those that have the power to shape discussions, so limiting speech is always counter productive.

advocate at least, absolutist is more of a slur

Those two are enormously different, though. I'd consider myself an advocate, just as anyone who believes in a fair and free democracy should. But I am very far from being an absolutist — and I have a secret suspicion that nobody actually is. Musk certainly isn't.

A few are and understand that most of the time your are defending scoundrels. But there is a sizeable and probable larger group that very easily wants to suppress speech they do not like. There never was a case where to much freedom of speech has been a significant problem, contrary to the other way around.

Next is misinformation and tomorrow you wonder why you cannot state your opinion anymore. A cycle that has been repeated ad nauseum. It just isn't a smart solution and causes more problems than it solves.

Do you get mad at google for automatically detecting and removing spam from your email inbox? For a lot of people, probably the majority, speech by scoundrels falls somewhere in that realm... there is simply no debate to be had about the basic humanity of certain classes of people. Capitalistic companies respond to this demand.

That said, I agree the government probably shouldn't be involved here for the most part (slippery slope, government is a blunt tool, etc.). As long as your "speech" isn't actually harming someone (harassment, revenge porn, incitement, etc.)

As long as we're defending scoundrels it's worth remembering we already lack so many protections for non-scoundrels. In a lot of states you can be fired if your boss hears a whiff of collective bargaining. But I digress.

We are not talking about spam if that wasn't a rhetorical question. Advertising not wanting any controversy attached to their product placement is no solution and isn't desirable. This isn't done in the name of users.

That there are limited worker protections in countries is a different problem, but is certainly not inhibited by too much speech, quite the contrary it would worsen the situation further. Civil liberties never suffered because too much speech was allowed, so the perspective to err on the side of freedom is only logical.

there is simply no debate to be had about the basic humanity of certain classes of people

That is just an invalid generalization.

Spam is an intractable problem for any so called free speech absolutist. One person's spam could be another person's desired message. But if a platform is overrun with spam, it becomes unusable for genuine discourse.

Maybe the biggest challenge is defining what constitutes "spam." While some cases seem clear-cut (e.g., repeated identical messages from bots, malware, phishing), others are quite subjective. Subtle marketing? Aggressive marketing? Repetitive but sincere advocacy for a cause? Repetitive but insincere trolling? Repetitive but sincere trolling?

All this seems rather obvious, so I was kind of surprised to see how many people bought into Elon's vision for Twitter, it was never workable.

Restricting speech is always in the interests of those that have the power to shape discussions, so limiting speech is always counter productive.

This is not true. Restricting hate speech is an obvious counterexample.

It isn't obvious at all. It doesn't help for that matter, but that is secondary. On the contrary, it is just a popular excuse to restrict speech because nothing about hate speech is objective. We see bad legislation around the globe and that will never protect any minority.

It is a bad idea and damaging and there is ample empirical evidence for that.

I love free speech, but you're going to have to convince me with a lot of evidence that Germany restricting pro-Nazi speech after WW2 was bad.

The solution to X (Twitter) sucking is to stop using it. It will either: get fixed or go out of business and be replaced.

It seems we've become a society that rewards bad practices with attention which is all any company on the web is trying to get, your attention.

Mastodon has been a breath of fresh air and you can get a really interesting feed going when you follow the right people and hashtags

And it can handle Twitter scale now

https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the...

Which people and hashtags? Trying to check it out but struggling to find relevant content. Is there a tech community somewhere? The ones I found appeared to be dead.