Hacker News

Top stories

Live mirror
30 storiesupdated just nowView source snapshot
  1. How to keep enjoying programming in a world of LLMs (haskell.org)
    5comments
  2. Revealing the details of how OpenAI agents hacked Hugging Face (swarmtraces.org)
    300comments
  3. A single function Jev-like wrapper for LLMs, including vision models (allanrbo.blogspot.com)
    18comments
  4. We're gonna need a lot more mathematicians (terrytao.wordpress.com)
    164comments
  5. Plan mode is dead (aymannadeem.com)
    304comments
  6. Ollaya – Ollama for open-source, Jev-style decision models (ollaya.dev)
    117comments
  7. Show HN: Jev Plays Pokémon Red (jev-pokemon.vercel.app)
    84comments
  8. What even is an OS now? (sockpuppet.org)
    283comments
  9. Is your Postgres migration safe or not safe? (safenotsafe.dev)
    8comments
  10. 16GB iPod Nano 3G Upgrade (tuckerosman.com)
    1comments
  11. Floci: Locally emulating any cloud service (floci.io)
    1comments
  12. The Murky History of Soviet-Born Tetris (mitpress.mit.edu)
    12comments
  13. Jury finds Facebook liable for deceiving users in Cambridge Analytica case (cbsnews.com)
    50comments
  14. Scientists build most accurate atomic clock (phys.org)
    4comments
  15. Gravity seems holographic. What does that mean for reality? (quantamagazine.org)
    171comments
  16. Ask HN: Who's still keeping a DOS machine up because the business depends on it?
    111comments
  17. Fourier Analysis: Drawing Llamas with Circles (adekau.github.io)
    3comments
  18. Excel now supports multiple values in a single cell (techcommunity.microsoft.com)
    138comments
  19. A new world airport and its baggage (computer.rip)
    2comments
  20. First Principles Thinking (sunilsadasivan.com)
    112comments
  21. HomelabFest will be in St. Louis in September 2027 (homelabfest.org)
    15comments
  22. One Piece of Flock Camera Data Put This Innocent Woman in Jail for 13 Days (jezebel.com)
    77comments
  23. I wrote a ray tracer in Brainfuck (epestr.com)
    17comments
  24. Show HN: Hacker Atlas - A map of what Hacker News talks about (hackeratlas.com)
    20comments
  25. From Thin Air to Bootable Images: The Tine Build System (amutable.com)
    —discuss
  26. Lab on a Contact Lens Can Measure Stress Through Serotonin (ieee.org)
    11comments
  27. Alberta's image as world's only rat-free region shattered by discovery of rat (theguardian.com)
    18comments
  28. Remembering Johannes Doerfert (llvm.org)
    3comments
  29. U.S. appeals court upholds designation of Anthropic as supply chain risk (cnbc.com)
    773comments
  30. Postgres SELECT DISTINCT Does Not Scale (dbos.dev)
    24comments

Postgres SELECT DISTINCT Does Not Scale

83 pointsby 1d agodbos.dev
24 comments
7h agoHN ↗

"Postgres SELECT DISTINCT Does Not Scale"

Correct. This is documented in depth: DISTINCT sorts the results first.

The article's use case seems to imply the author did not know about GROUP BY, nor does it imply the author knew about indexes, nor ANALYZE. Postgres 18's new skip scan indexing also could help here, so ensuring the planner chooses that could help.

7h agoHN ↗

Would GROUP BY fix the issue?

The article explains that skip scan doesn't do anything here.

nor does it imply the author knew about indexes, nor ANALYZE

Indexes were talked about a lot, and they explicitly mentioned looking at the query plan.

4h agoHN ↗

nope - either gets you a HashAggregate, GroupAggregate, or Unique depending on whether the column/input is indexed/ordered

4h agoHN ↗

The article seems to have changed since I commented.

3h agoHN ↗

Why don't he use GROUP BY on indexed columns was my first thought also.

I guess we just have to patiently wait for the OP to hopefully read Postgres documentation from postgreSQL 10.x before correcting the article again.

3h agoHN ↗

Does that fix it? Why would people be trying to add new index scanning modes if that's enough to fix it?

Someone else said it doesn't.

18m agoHN ↗

Why don't he use GROUP BY on indexed columns was my first thought also.

Because it has the same issue.

SELECT DISTINCT and GROUP BY are equivalent from a query planner perspective.

5h agoHN ↗

Did you even read the article? They show that a perfect index for their query didnt help because Skip Scan is currently not used for DISTINCT queries.

4h agoHN ↗

The article seems to have changed since I commented.

3h agoHN ↗

It says that the company is co-founded by Postgres creator. I find that bit hard to believe given that there is nothing novel in the article, probably discovery for them. I do understand that everyone has to go through their own journey to learn these things but at the same time when you are running business then seeking professional help isnt a bad idea.

Based on my experience queries like these cannot scale, whatever you do. However if you are already on a path where you had invested a lot in such queries then hire a DBA, if you are not far off then hire an architect to model the data for better performance.

3h agoHN ↗

Scale with what? If you have m distinct values in an index, then listing them this way takes m log(n) time, which is fine for many use cases no matter how much data you have.

3h agoHN ↗

The way the OP is trying to achieve all the goals by pushing the complexity on the queries/database is what I am referring to as non-scalable as data grows on SQL DB.

If you have m distinct values in an index, then listing them this way takes m log(n) time, which is fine for many use cases no matter how much data you have.

And NO the runtimes are not right away applicable on machines at scale. You are dealing with DB locks, page sizes, available memory, existing data in memory, queue depth. Experienced folks get paid to short circuit such learnings

3h agoHN ↗

Runtimes are usually pretty well applicable at scale, it’s just that most people don’t have a good intuition about asymptomatic notation. Constants and lower order terms matter a lot in practice but are hidden in asymptotic notation.

2h agoHN ↗

Selecting distinct values of a single column with a simple condition is hardly pushing complexity into the database.

3h agoHN ↗

It's Stonebraker, he has a history of doing this to sell shit to people who don't need it.

2h agoHN ↗

I've generally started to treat use of SELECT DISTINCT as a warning flag, as it's very common that it indicates bad code

Some use it because they don't understand uniqueness constraints and try to fix it in post so to say. Some use it because they forgot a join condition and are absolute amateurs. Some use it because it fixed a problem for them once and now they add it everywhere

These people seem to outnumber the people who use SELECT DISTINCT in a well thought out manner

2h agoHN ↗

It's definitely a warning flag if you're applying it to full-ish rows. This situation seems much more innocuous to me.

1h agoHN ↗

Yeah, people have been advising against DISTINCT for ages. I guess that piece of common wisdom somehow got lost in this age of AI wonders. :D

1h agoHN ↗

Well, if the solution is a manual workaround that forces a better query plan, this needs to be a part of Postgres itself so it builds better query plans automatically.