I sort of think there should be an 'important flagged discussions' link, ones with otherwise lots of engagement that got flagged by people who were opposed to the viewpoint of the post.
I believe it will show it, but not the flagging specifically. You can see the position (best) drop off a cliff, even while upvotes and comments keep rising for a bit.
Maker, this is really really cool. Only quibble is make it easier to check out a post. Not sure how, but maybe:
- Bookmarklet
- Some sort of search, or bring up all posts by a user, or all posts currently on the front page.
A suggestion I have is to allow a comparison chart. So be able to add multiple submissions, and compare their trends.
This view of HN more heavily weights articles that generate robust discussions. I often find interesting stuff here that only briefly made it to the front page (due to flags or due to HN's "flamewar detector").
I believe that HN is intervening manually. They can give it a up or a down push.
Perhaps I could use the data to reverse engineer what is going on behind the scenes.
Neat! I wonder how it works. Does it poll every submission every set amount of seconds to check key stats over time? Looks like a huge amount of traffic. Is there any more efficient way I'm not seeing?
I would think that it loads page one and two of new and best every minute and records submissions. That is 4 requests per minute, which isn't that bad.
Does it track things that were on the front page but were flagged? I've been looking for that.
...and...
I sort of think there should be an 'important flagged discussions' link, ones with otherwise lots of engagement that got flagged by people who were opposed to the viewpoint of the post.
This article was on my frontpage, I read the 10 or so comments that were there and then forwarded it to myself as I love reading discussions about culture war topics, but when I refreshed HN immediately after reading the comments, I discovered that not only was it no longer on the front page, but it couldn't be found on any of the 17 pages provided when hitting next and searching for it. This is far from the first time I've noticed this happening with articles of this nature, and I think it would be very interesting to be able to see what ideas the HN moderators don't want the minds of their users exposed to.
I think there are also several specific (and influential) subreddits that would also yield some very interesting results. Of course, you run the risk of getting your scraper blocked which would ruin your entire project...but on the other hand, if you were able to find a technical way around such blocking (I assume there are services that allow one to scrape using a large pool of ip's?) I think you could make this project significantly more interesting than it already is. Being able to see the things you are already showing is mildly interesting, but being able to see a list of particular topics that the powerful gatekeepers of information (in a very real sense, demi-gods who exert significant control over the publics' very perception of reality) do not want people to see is far more interesting, and in turn may make your service far more popular than it would be in its current form.
EDIT: And it keeps getting more interesting...now the parent comment to the one I made[1] in that thread is [deleted] - not sure if that's a user invoked delete or moderator - I assume the former, as if it was a moderator I assume mine would have been toasted also.
> Even re-production and communication should be discussed.
I think the most interesting and valuable/useful perspective to discuss this topic from (aside from "the science") is from a human psychology (which is heavily dependent on communication) perspective. Ultimately, actually doing something about climate change (regardless of what the consensus decision of that is) requires political support, and from what we've seen on that front, that seems to be the weakest link within the entire multidimensional problem.
Speaking of human psychology (and communication, and perception of reality), note how quickly moderators were able to memory hole this post - it was on my front page 10 minutes ago, and now it can no longer be found within the first 17 pages of HN. I've seen this phenomenon occur on several occasions over the years, and I don't even check HN all that often, and in each case (of the ones I've noticed) it tends to be particularly controversial topics like this one, where the perspective of the article is "controversial".
Human beings (and all the clever mischievousness they get up to) are completely fascinating creatures, I can't understand why so few people find them interesting.
EDIT 2: And just now does the post have a [flagged] tag, wonder why it disappeared before even being flagged.
There is good research to be done throughout social media on flag/dupe/downvote trends and countermeasures, whether inspired by the zeitgeist, unpaid advocates for a viewpoint, paid reputation management firms and their clients, or social-bot experiments on influence.
Agreed...going even further, I think ~collective observation, analysis, and self/meta-reflection of the internet hivemind/zeitgeist/simulation/matrix could be a huge new internet platform space...and not just popular and fun/interesting, but I truly believe that the near total absence of skilled self-reflection is just what the doctor ordered for the period of social and political polarization that we find ourselves in. I think once people "got" the idea, it would catch on like wildfire (I also think that if a nefarious actor got there first (Facebook, Google, etc), they could possibly capture and nullify the space).
I know of absolutely zero startups that are active in this space, about the closet thing I've come across are platforms like https://www.kialo.com/ that are trying to improve the quality and structure of conversation.
Do you know of anything along the lines of what I'm describing here (assuming it makes sense what I'm describing)?
Do you know of anything along the lines of what I'm describing here
I'm only aware of organizations that have a business incentive to influence at-scale sensemaking or decisions, so their analysis is private, and their public participation is often unattributable and oriented towards a specific side/outcome. Examples could include Cambridge Analytica, public-private partnerships on semi-automated data analytics for psychological warfare, influence operations and persona management, and PR firms (http://www.techsoc.com/grassroots.html).
As for consumer-accessible data visualization and models for interactions within specific communities, it's the eternal challenge of gamified semantics in any sufficiently-contested online space, with Google search rankings/algo/gaming being one of the most visible battlegrounds. Every meta-observation and subsequent algo/ranking update leads to new outcome (e.g. profit) variability, which changes the live experiment and starts a new iteration of the ongoing game.
The most successful historical examples have been some decentralized combination of human moderation, participant voting and usually-private analytics tools available to moderators. Over time, we've seen the most successful communities move towards increased transparency of manual moderation. Independent, adversarial analytics (e.g. hnrankings or this tool) provide valuable feedback to both users and moderators.
Kialo looks fun, thanks for the pointer. Do you know their business model?
When online communities desire other forms of government, such as ones that take many members’ opinions into account or that distribute power in non-trivial ways, communities must resort to laborious manual effort. In this paper, we present PolicyKit, a software infrastructure that empowers online community members to concisely author a wide range of governance procedures and automatically carry out those procedures on their home platforms ... We demonstrate the expressivity of PolicyKit through implementations of governance models such as a random jury deliberation, a multi-stage caucus, a reputation system, and a promotion procedure inspired by Wikipedia’s Request for Adminship (RfA) process.
I'm only aware of organizations that have a business incentive...
Ya, that's a huge part of the problem in the world today, making the world a better place typically doesn't pay too well (despite there being potentially massive value realized to society), but engaging in harmful activities often pays nicely. And on top of it, our governments seem to be fairly clueless, if they even care beyond political opportunism.
The most successful historical examples have been some decentralized combination of human moderation, participant voting and usually-private analytics tools available to moderators. Over time, we've seen the most successful communities move towards increased transparency of manual moderation. Independent, adversarial analytics (e.g. hnrankings or this tool) provide valuable feedback to both users and moderators.
Ya, extremely well thought out self-moderation towards establishing of a very deliberate community culture is something that I would like to see tried, all within a community who's goal is not just having fun discussing some niche topic, but rather with the explicit goal of analyzing at scale the collective behavior (and the underlying thinking that powers it) within our increasingly chaotic societies. Maybe I'm excessively pessimistic, but I look at the conversations I see online (including here at HN, in threads on culture war topics) and it makes me very concerned for where this world is going to be 10 to 30 years from now.
Kialo looks fun, thanks for the pointer. Do you know their business model?
No idea....developing a functional business model for such a community would probably be much harder than normal...but then there shouldn't be a need for money beyond eventual wages and operational expenses...the value is in what it would bring to society, and I believe there are plenty of people out there still that can appreciate the importance of such ideas.
I did that for reddit in 2016. Especcially with /r/the_donald. Funny to watch the steep hill for every upvoted thread after 30 mins or so: 5 minutes of massive bot action with 2000-5000 upvotes and than flat again.
Does it track things that were on the front page but were flagged? I've been looking for that.
I sort of think there should be an 'important flagged discussions' link, ones with otherwise lots of engagement that got flagged by people who were opposed to the viewpoint of the post.
Flags are for "this shouldn't be here" not for "I disagree", AFAICT. It's bad enough that downvotes are used for disagreement.
I'm pretty sure that flags are used for disagree with viewpoint of this article.
on edit: discussing how I see flagged used, not how it should be used.
People who do that are supposed to lose their flagging privileges.
I believe it will show it, but not the flagging specifically. You can see the position (best) drop off a cliff, even while upvotes and comments keep rising for a bit.
Maker, this is really really cool. Only quibble is make it easier to check out a post. Not sure how, but maybe:
- Bookmarklet
- Some sort of search, or bring up all posts by a user, or all posts currently on the front page.
A suggestion I have is to allow a comparison chart. So be able to add multiple submissions, and compare their trends.
All in all, great work!
It would be cool to see what popular posts are flagged off the homepage.
You can often find these by going to: https://news.ycombinator.com/active
This view of HN more heavily weights articles that generate robust discussions. I often find interesting stuff here that only briefly made it to the front page (due to flags or due to HN's "flamewar detector").
Demo: https://upvotetracker.com/post/hn/27092770 .
The link shows voting history of this submission. I guess it shouldn't be a surprise that it works on itself; still, I find it very cool that it does.
What happened at 7:36UTC? It dropped from second place to 16th.
I believe that HN is intervening manually. They can give it a up or a down push. Perhaps I could use the data to reverse engineer what is going on behind the scenes.
This is a cool tool. Nice work. Fun to play with. Also, I think it could benefit yet another hacker trying to hack HackerNews.
An earlier effort: http://hnrankings.info/
Neat! I wonder how it works. Does it poll every submission every set amount of seconds to check key stats over time? Looks like a huge amount of traffic. Is there any more efficient way I'm not seeing?
I would think that it loads page one and two of new and best every minute and records submissions. That is 4 requests per minute, which isn't that bad.
Currently I am doing 6 requests a minute
I wonder, might you consider implementing something like what's asked for here?
https://news.ycombinator.com/item?id=27093483
...and...
An example of such a post:
The not-so-drastic truth behind climate change
https://news.ycombinator.com/item?id=27097434
This article was on my frontpage, I read the 10 or so comments that were there and then forwarded it to myself as I love reading discussions about culture war topics, but when I refreshed HN immediately after reading the comments, I discovered that not only was it no longer on the front page, but it couldn't be found on any of the 17 pages provided when hitting next and searching for it. This is far from the first time I've noticed this happening with articles of this nature, and I think it would be very interesting to be able to see what ideas the HN moderators don't want the minds of their users exposed to.
I think there are also several specific (and influential) subreddits that would also yield some very interesting results. Of course, you run the risk of getting your scraper blocked which would ruin your entire project...but on the other hand, if you were able to find a technical way around such blocking (I assume there are services that allow one to scrape using a large pool of ip's?) I think you could make this project significantly more interesting than it already is. Being able to see the things you are already showing is mildly interesting, but being able to see a list of particular topics that the powerful gatekeepers of information (in a very real sense, demi-gods who exert significant control over the publics' very perception of reality) do not want people to see is far more interesting, and in turn may make your service far more popular than it would be in its current form.
EDIT: And it keeps getting more interesting...now the parent comment to the one I made[1] in that thread is [deleted] - not sure if that's a user invoked delete or moderator - I assume the former, as if it was a moderator I assume mine would have been toasted also.
https://news.ycombinator.com/item?id=27098186
Human beings (and all the clever mischievousness they get up to) are completely fascinating creatures, I can't understand why so few people find them interesting.
EDIT 2: And just now does the post have a [flagged] tag, wonder why it disappeared before even being flagged.
There is good research to be done throughout social media on flag/dupe/downvote trends and countermeasures, whether inspired by the zeitgeist, unpaid advocates for a viewpoint, paid reputation management firms and their clients, or social-bot experiments on influence.
Agreed...going even further, I think ~collective observation, analysis, and self/meta-reflection of the internet hivemind/zeitgeist/simulation/matrix could be a huge new internet platform space...and not just popular and fun/interesting, but I truly believe that the near total absence of skilled self-reflection is just what the doctor ordered for the period of social and political polarization that we find ourselves in. I think once people "got" the idea, it would catch on like wildfire (I also think that if a nefarious actor got there first (Facebook, Google, etc), they could possibly capture and nullify the space).
I know of absolutely zero startups that are active in this space, about the closet thing I've come across are platforms like https://www.kialo.com/ that are trying to improve the quality and structure of conversation.
Do you know of anything along the lines of what I'm describing here (assuming it makes sense what I'm describing)?
I'm only aware of organizations that have a business incentive to influence at-scale sensemaking or decisions, so their analysis is private, and their public participation is often unattributable and oriented towards a specific side/outcome. Examples could include Cambridge Analytica, public-private partnerships on semi-automated data analytics for psychological warfare, influence operations and persona management, and PR firms (http://www.techsoc.com/grassroots.html).
As for consumer-accessible data visualization and models for interactions within specific communities, it's the eternal challenge of gamified semantics in any sufficiently-contested online space, with Google search rankings/algo/gaming being one of the most visible battlegrounds. Every meta-observation and subsequent algo/ranking update leads to new outcome (e.g. profit) variability, which changes the live experiment and starts a new iteration of the ongoing game.
The most successful historical examples have been some decentralized combination of human moderation, participant voting and usually-private analytics tools available to moderators. Over time, we've seen the most successful communities move towards increased transparency of manual moderation. Independent, adversarial analytics (e.g. hnrankings or this tool) provide valuable feedback to both users and moderators.
Kialo looks fun, thanks for the pointer. Do you know their business model?
Here's a recent paper on community governance automation: https://www.csee.umbc.edu/courses/graduate/691/spring21/pdf/...
Ya, that's a huge part of the problem in the world today, making the world a better place typically doesn't pay too well (despite there being potentially massive value realized to society), but engaging in harmful activities often pays nicely. And on top of it, our governments seem to be fairly clueless, if they even care beyond political opportunism.
Ya, extremely well thought out self-moderation towards establishing of a very deliberate community culture is something that I would like to see tried, all within a community who's goal is not just having fun discussing some niche topic, but rather with the explicit goal of analyzing at scale the collective behavior (and the underlying thinking that powers it) within our increasingly chaotic societies. Maybe I'm excessively pessimistic, but I look at the conversations I see online (including here at HN, in threads on culture war topics) and it makes me very concerned for where this world is going to be 10 to 30 years from now.
No idea....developing a functional business model for such a community would probably be much harder than normal...but then there shouldn't be a need for money beyond eventual wages and operational expenses...the value is in what it would bring to society, and I believe there are plenty of people out there still that can appreciate the importance of such ideas.
It is a web crawler that goes through the front page and new submission page every minute.
Neat, it feels fast and the charts are beautiful!
T̶h̶e̶r̶e̶ ̶s̶e̶e̶m̶s̶ ̶t̶o̶ ̶b̶e̶ ̶a̶n̶ ̶i̶s̶s̶u̶e̶ ̶w̶i̶t̶h̶ ̶t̶h̶e̶ ̶u̶p̶v̶o̶t̶e̶s̶ ̶t̶h̶o̶u̶g̶h̶:̶ ̶F̶o̶r̶ ̶e̶x̶a̶m̶p̶l̶e̶,̶ ̶t̶h̶e̶ ̶n̶u̶m̶b̶e̶r̶ ̶o̶f̶ ̶u̶p̶v̶o̶t̶e̶s̶ ̶o̶f̶ ̶t̶h̶i̶s̶ ̶s̶u̶b̶m̶i̶s̶s̶i̶o̶n̶ ̶i̶t̶s̶e̶l̶f̶ ̶a̶t̶ ̶6̶:̶2̶0̶:̶0̶8̶ ̶a̶m̶ ̶s̶h̶o̶u̶l̶d̶ ̶b̶e̶ ̶2̶9̶ ̶i̶n̶s̶t̶e̶a̶d̶ ̶o̶f̶ ̶2̶ ̶(̶I̶ ̶k̶n̶o̶w̶ ̶i̶t̶ ̶b̶e̶c̶a̶u̶s̶e̶ ̶I̶'̶m̶ ̶w̶o̶r̶k̶i̶n̶g̶ ̶o̶n̶ ̶a̶ ̶s̶i̶m̶i̶l̶a̶r̶ ̶t̶o̶o̶l̶ ̶:̶)̶)̶.̶
Edit: I was wrong, the upvote count is correct. I thought the timezone was UTC, it would be nice if the timezone information is added to the chart.
This is amazing. Would you consider releasing the source?
If not would you care to comment on development; language(s), dependencies etc?
Yes. Fantastic project! Woul love to know more details & key architectural decisions.
Perhaps I will do an article on it, I came up with some ingenious solutions.
Thank you! That'd be immensely helpful.
I am working on a similar concept as a side side project that progress way too slowly. Kudo for the nice implementation !!
Why doesn’t it find its own URL? Getting a 404 when searching for this very post.
You must enter the HN URL. Like this: https://news.ycombinator.com/item?id=27092770
I did that for reddit in 2016. Especcially with /r/the_donald. Funny to watch the steep hill for every upvoted thread after 30 mins or so: 5 minutes of massive bot action with 2000-5000 upvotes and than flat again.
If the graphs are publicly visible anywhere it'd be interesting to have a look