Hacker News

Top stories

Live mirror
30 storiesupdated just nowView source snapshot
  1. Claude Code now reads AGENTS.md if there is no Claude.md(claude.com ↗)
    128comments
  2. Android 17 is the first since 3.x to add new APIs without releasing to the AOSP(grapheneos.social ↗)
    177comments
  3. Saving another 100TB of RAM(cloudflare.com ↗)
    33comments
  4. Cloudflare Quick Tunnels(cloudflare.com ↗)
    218comments
  5. Xcode 27.1 Beta Release Notes(developer.apple.com ↗)
    57comments
  6. Cache-to-Cache: Direct Semantic Communication Between LLMs (2025)(arxiv.org ↗)
    11comments
  7. Photon-Emission-Guided Laser Fault Injection Enables RP2350 Secure Debug(ledger.com ↗)
    43comments
  8. US troop deaths during Iran war exceed Pentagon count by at least four(reuters.com ↗)
    5comments
  9. Show HN: Cactus Needle 3: 8-29MB automation models can match DeepSeek V4 Flash(cactuscompute.com ↗)
    71comments
  10. How to Write with an LLM(sockpuppet.org ↗)
    238comments
  11. OpenJev(openjev.com ↗)
    235comments
  12. The first new cat species discovered in 100 years(nationalgeographic.com ↗)
    34comments
  13. Cyclomatic Complexity in C#(ndepend.com ↗)
    9comments
  14. Two parallel neural ectoderm progenitors contribute to the developing brain(newscientist.com ↗)
    50comments
  15. The Implications of Linguistic Illegibility for LLM Security(arxiv.org ↗)
    16comments
  16. From Geometry to Algebra and Back Again: 4000 Years of Papers (2023) [video](youtube.com ↗)
    discuss
  17. C++26: Trivial infinite loops are no longer undefined behaviour(sandordargo.com ↗)
    166comments
  18. Size-Specialized Memory Allocation(go.dev ↗)
    3comments
  19. Minimal Phone 2(minimalcompany.com ↗)
    142comments
  20. How SpaceX streamlined the Raptor engine(construction-physics.com ↗)
    23comments
  21. Warez: The Infrastructure and Aesthetics of Piracy (2021)(archive.org ↗)
    12comments
  22. I vibed a proof of Conway's conjecture(overreacted.io ↗)
    174comments
  23. Inside ZCode: Silently uploading your Git history to the cloud(ferstar.org ↗)
    89comments
  24. A search-and-inference database from scratch in pure Zig(antfly.io ↗)
    16comments
  25. Korea raises data breach fines to 10% of revenue(koreajoongangdaily.com ↗)
    70comments
  26. Border agents can search cellphones without a warrant or reasonable suspicion(lawandcrime.com ↗)
    130comments
  27. Cekura (YC F24) Is Hiring(ycombinator.com ↗)
    discuss
  28. US Military had close call after using AI for hallucinated intelligence report(cnn.com ↗)
    275comments
  29. Mathematicians Build Long-Awaited Graph Sandwich(quantamagazine.org ↗)
    15comments
  30. North Korean nuclear test sets off years of earthquakes(science.org ↗)
    147comments

Why's that company so big? I could do that in a weekend

712 pointsby 10y agodanluu.com
423 comments
10y agoHN ↗

I personally have never seen this complaint leveled against the mighty Goog, nor really any company that's cash positive. When I see people giggle at bloat, it's more often than not twitter and still more often than not, it's not intrinsically because big company = waste, but because company that should be small is made to be big either by VCs or Wall Street, and then can't turn a profit anymore.

No, maybe if twitter was still a tiny company you couldn't use it in Arabic but maybe they also wouldn't constantly be on the edge of going out of business.

Edit: I think this is also a regular criticism of Twitter because Twitter has spent the last several years trying to Facebook itself and strayed far from the original idea. You can argue the pros and cons all day since they're pretty much infinite on both ends, but Twitter as a company has spent vast amounts of money on a lot of features that none of it's original userbase really wanted (and drove a lot of people off in the process).

Me, I only really ever used it as a time-waster so I'm not awfully attached but I listen to a ton of podcasts and I can't count the number of hosts who disliked more or less every new thing Twitter did.

10y agoHN ↗

Right, I feel like the author has a good angle with which to tackle some of this criticism, but choosing Google as the example company is really unpersuasive for me. I'd agree that Twitter would be a better example (or Uber, 2,000 engineers - doubled in size over the last year)

10y agoHN ↗

Uber need their 2000 engineers to write their 1000 microservices eh?

<snark/>

10y agoHN ↗

Joking aside I enjoyed that video (https://www.youtube.com/watch?v=kb-m2fasdDY - for those who didn't see it), the presenter was pretty honest and way more open than I expected. I'm just so curious how those services break down - if it's genuinely on the "lstrip as a service" level or whatever, whether that's "1,000 services" figure was actually a relatively simple API with 10 versions at various levels of deprecation, whether he meant it was 1,000 different service endpoints (250 services each with POST, GET, PUT, DELETE) or what. I want to know more basically :)

10y agoHN ↗

I am usually not someone who falls into the "what do they need so many people for, it should be simple"-trap but I was still surprised to learn that Lyft has 400 engineers and Twilio has around 300 too. I am note sure they would be that big if they had to finance everything out of their profits, but i guess it's a good way to maximize progress, on the other hand they also had a lot of challenges with growing dev teams.

10y agoHN ↗

Things like Dropbox are striking. I guess it was mostly written originally by Drew Houston over a year or so and was cool.

Then 2011, 50 million users 70 employees. Seems fair enough.

Now about 1500 employees. It worked fine and scaled fine with 70 employees so you wonder if hiring the other 1430 was good business. I guess if VCs are throwing money at you you may as well spend it on something. Though that's probably what Yahoo was thinking when it went to 11,700 employees and that didn't play out so well.

10y agoHN ↗

I bet its mostly sales. I had no idea about how much sales ppl you need at certain scale.

10y agoHN ↗

Sales persons are necessary when continuing growth requires efforts that do not scale as effectively. Marketing and advertising campaigns work sufficiently well for b2c with 1-2 people but you hit a wall if you want to selling to, for example, the Pentagon because the way those with the deepest pockets buy is nothing like how most efficient individuals and organizations buy. Groupon's massive needs for salespeople is an example of how even lower margin b2b even needs sales due to how scaling out b2b is so much less cost-effective. But for investors, landing a huge contract is not as much about the revenue as much as an indicator of stability in terms of both revenue and market signal (no Fortune 100 will buy some random fly-by-night vendor's stuff without code escrow even if it's the best thing ever because their sourcing overhead is so poor and big companies are immensely risk-averse by design).

10y agoHN ↗

Atlassian's 'b2b without salespeople' approach has worked well for them.

10y agoHN ↗

They definitely have sales roles in more recent years and have a sales ops team [1][2][3]. The Federal channel manager description even acknowledges that there's unique challenges in selling to the government that necessitates someone that knows the different contracting vehicles. What they've done with little turn to sales culture is pretty admirable but at the same time only possible with organic, engineer-to-engineer driven growth Github and Atlassian suites.

Salesforce grew somewhat similarly via rogue accounts on expense accounts, but to actually clinch the biggest growth there was no way to avoid salespeople to get over $500MM in revenue probably.

[1] https://www.smartrecruiters.com/Atlassian/99086565 [2] https://www.smartrecruiters.com/Atlassian/95395260 [3] https://www.smartrecruiters.com/Atlassian/93068238

10y agoHN ↗

Yea, looking at the employee page for Stack Overflow shows a vast majority of the staff being in sales.

10y agoHN ↗

Well VCs want to know what their money does. So expect a large part of controlling.

10y agoHN ↗

I think those people miss the point that not all engineers are working on the core products.

For example, as one grows they need to have a dedicated HR teams. These teams will demand for a HR software to make their life easier. If the software is an on premise solution then you require engineers to manage that software. Most HR systems require a RDBMS backend, so there is an additional need for a DBA. As this adds to the company, there is a need for a hosted Identity Management solution, which again requires dedicated engineers...so and so forth.

While there are many reasons for a company to be loosing money and most of them are related to building a sustainable business, from an engineering point of view it has a lot to do with hiring. Companies with lot of cash want to hire talented engineers and there is nothing wrong with that when getting the company off the ground. But as they grow the talented pool they can hire from gets smaller. So either they start overpaying for a great talent or offering above market rates for a mediocre talent. Its mostly the latter people who offer negligible intrinsic value while getting boat loads of money.

10y agoHN ↗

I'm definitely not claiming to know what's good/bad for a company I've never worked for, I'm saying the perception from outside is that companies hit these sizes, hire massive amounts of people to do something, and then the products almost universally become worse.

Maybe it's just big-company-itis.

10y agoHN ↗

What features has Twitter shipped? Moments??? Pfft.

On the contrary I'd argue they've hardly done anything new, and that is why they are struggling. Think of all the things Facebook has done and abandoned in the same time.

But I like Twitter - I just wished they fixed some pretty obvious issues. I mean... Tweetstorms are like a request for a feature right there, and Twitter's response is... silence.

10y agoHN ↗

As a counter point, on every product I've worked on, on every time we've wanted to make a 'big leap' we've need to get a small expert team to go make it happen.

10y agoHN ↗

Dod that small expert team polish the product and maintained it for the next 2 years?

Something about 80%/20% rule...

10y agoHN ↗

And yet sometimes a small team really could do it. I spent years complaining about MSN/AOL/Skype etc. and talking about how a small team could build a better messenger in a week, but we didn't really believe it. Then Slack actually took that possibility seriously, and look at the results.

10y agoHN ↗

Slack also jsut recently hit 3 million users. 100 million people were using AOL Messenger at it's peak, and 12 million were using ICQ when AOL bought Mirabillas.

10y agoHN ↗

Yes, but that's something a lot of people are arguing here and I agree. The "I could build in a week" is ridiculous, but "I could what this 1000 engineers company do that with a small team of 10, in a lean way in 18 months" is very frequent in the market.

10y agoHN ↗

MSN and AOL come from the 90s, and Skype is from 2003. Slack is from 2013 - the web has more than doubled in age for Slack compared to the others. Computers are ridiculously faster, browsers are more mature, language libraries are more complete.

Slack also took much longer than a week, and to be honest, the actual chat part of slack is pretty meh. Where Slack unbelievably shines is in the buttery-smooth onboarding process and the management around the chat accounts.

10y agoHN ↗

I think a lot of people are willing to "leave cash on the table" if it means a smaller, more effective team and leaner product.

10y agoHN ↗

Yup. Have never gotten the whole "Airbnb/Uber/<startup>" is just a crud app line of thought.

If they're crud apps then everything else in between is just glue programming.

The author of the post covered some very astute points but one not covered in much detail is the challenge of scale and reliability. Maybe this falls under the bucket of optimization and/or latency but I think this deserves being called out on its own.

Once you have Uber-scale number of users requesting taxis at any point, and have a network of drivers constantly communicating their position with the app, suddenly you no longer have a trivial crud app on your hands...

10y agoHN ↗

These companies are not "tech" companies in my book any more than the Sears catalog was a "tech" innovation.

They are normal business that happen to be enabled by technology, but the technological challenges are not new or to be frank uncommonly challenging.

Because we're I assume mostly coders, we think they are differentiated by tech, almost everything else (marketing, sales, customer service) is likely more important.

Bad technology could certainly kill a platform, but excellent tech won't save it, and a mediocre implementation is good enough.

10y agoHN ↗

Exactly. I would be surprised if Uber hired more than 20 developers (which I consider a lot already).

10y agoHN ↗

I would extremely impressed if Uber is maintained and developed by only 20 developers.

10y agoHN ↗

Just FYI I recently watched a tech talk from some conference where Uber guy claiming they have 2000 developers...

10y agoHN ↗

Does that include everybody working on autonomous driving and related 'side projects'?

10y agoHN ↗

I agree with you and parent that it is (and should) be a > 20 person job for its scale, but the career page is probably not the best example.

Many of those positions are perpetually open or rotated periodically (see: "You're leaking trade secrets" by Michael Schrenk) while the actual position was given to someone internal or someone connected to someone else internal to the firm.

Most of these pages are used for resume harvesting so that [if the firm has resources] an HR or technical recruiter can periodically evaluate the rest of the field. The firms holding out for the "rock star" often are more aggressive in recruiting them (whether at Job fairs or through LinkedIn headhunting).

10y agoHN ↗

They have an Android app, an iOS app, driver management system, fleet management system, ride tracking system, ride scheduling, navigation system (not sure if this is 3rd party or in-house), payment processing, profile system for both riders and drivers, price calculation system, and probably a bunch of other stuff in the background.

All this must work correctly and reliably in the 400+ cities they're operating in. To me at least, 20 developers is not enough to develop and maintain that many interconnected systems. I would say 100s at the very least.

10y agoHN ↗

20 people is more than enough to build and support these things...

10y agoHN ↗

Are you serious? With many 9s of uptime, low latency requirements and hundreds of millions of dollars in revenue on the line? Uber employs over 2k developers right now.

10y agoHN ↗

Don't forget cars that drive themselves

10y agoHN ↗

I'm not sure just because you mentioned 400+ cities they need as many developers as cities they operate in.

10y agoHN ↗

Did you read the article? In it, the author discusses the engineering issues that arise from trying to deal tackling problems in different languages. Now, granted, Uber in this particular thread of discussion does not have to deal with tokenization issues of Japanese/Chinese character search strings but I think it is naive to not think that the nuances of individual communities of 400+ cities intra-country, let alone dealing with separate nations, will add significant engineering overhead... (even if the relationship is not linear)

10y agoHN ↗

If you think scaling a service like Uber is not challenging, you're vastly underestimating how large they are.

Uber especially has requirements that make it tough to scale correctly. You need to track each Uber session in near real-time, right? That means you simply cannot afford to drop or interrupt those sessions. It's a completely different problem than scaling Instagram or WhatsApp.

10y agoHN ↗

Hmmm... nothing like near real-time. Challenging near real-time is sub-nanosecond.

Uber real-time requirement of what < 30s? (for a viable product).

No that doesn't seem like a challenging problem. It seems like a problem that would require almost no throught for a MVP, and then incremental challenges as the problem scales.

It seems like a problem that is so comparatively easy compared to what computers are capable of that I wouldn't even class it as "difficult".

But what's telling about your answer is that your thinking about this technical challenge, this feature as being of primary importance.

Do you think if Uber was 10 times cheaper than a regular taxi real-time tracking would be as significant?

Uber's funding which has enabled them to subsidize their expansion (and rides), advertising and promotion and possibly that they plugged in to a easily accessed communications channel (mobile apps). Were likely all more important than the quality of their implementation.

10y agoHN ↗

Probably less than 30s between pings during a ride, but I haven't tested it tbh. I usually use the word real-time when talking about systems that operate in the ns time range, and near real-time when talking about software. Sorry for the confusion.

What I was trying to emphasize is that Uber needs to keep track of these sessions and make sure they're not interrupted, or else they lose customers.

Yes, I know building something that fits these requirements and works most of the time is trivial. But we're talking about a system that must basically work as close to always as possible. More importantly, it must do this at scale, maintaining and keeping track of perhaps millions of sessions at once.

In a nutshell, an Uber MVP is probably not difficult, but the transition from MVP to real-world product is non-trivial in my opinion.

I'm actually planning on setting up an Uber-like service for my home country, which is why I've been giving these issues some thought.

10y agoHN ↗

Even 1M simultaneous users reporting every ~30 seconds could average down to ~30k messages per second. And they'd be extremely trivial to segment between servers. You're right, not a weekend hack, but a few weeks of a small team's time isn't out of the question. The technical side is not the challenge here though really - the business side is.

10y agoHN ↗

Perhaps you're right when it comes to scaling, but you're assuming that the system was designed correctly in the first place, which would be impressive imo. There is also a lot of failure handling and recovery that needs to be done gracefully, which I think would be complicated for a session-based system.

Again, I'm no expert, just making educated guesses!

10y agoHN ↗

I'm not sure why you need near real-time single session for the tracking. A ping from the app every couple seconds for the available driver screen. Then for the route tracking and billing have the same pings include the GPS track that the phone is keeping track of. If the servers lose track of any of the cars the data would be stale until it comes back into connection.

Also Uber is pretty conveniently partitionable. There's a little bit of overlap in some places but that's mainly for the available drivers screen.

10y agoHN ↗

I agree with you completely. I think a lot more effort is required once you are running the system at scale. Since I don't have experience with running Uber, I'm simply making an educated guess as to what kind of complexity is involved.

10y agoHN ↗

In addition the whole "we code for a better world" is just HR marketing to attract engineers. Sure there are companies that have innovative areas and oppurtunities, but many engineers are just code monkeys fullfilling daily duty without glory.

10y agoHN ↗

I read an article the other day that claimed Uber was beating Lyft because they had a strategy of opening a physical office in each city, having an office for drivers to sign up, and a human being who tailored things for that location. Their rival on the other hand assumed that some programmers in SF could do everything neccessary from a distance. Not sure how true it is, but it seemed an interesting theory.

10y agoHN ↗

Most apps are crud at the end of the day. Making them scale is the hard part.

10y agoHN ↗

Well, that's the thing - they are just a CRUD app in a lot of ways. Even at "uber scale" it really isn't that bad, it'd be a few incremental challenges but nothing a small team couldn't handle from a technical perspective. The catch though is that on top of that CRUD app is a business which involves some serious legal and marketing divisions to make that app work.

The tech may be simple, but the business side of it certainly isn't. The "I could hack that up in a weekend" attitude is often true - but it only covers the bare basics of the software it doesn't even attempt to address the business, which if often where the real struggle is these days. Especially if your background is development. I think many engineers still say this knowingly though, just to (perhaps incorrectly) scoff at the value of good business people.

10y agoHN ↗

I recently had a discussion about how hard it would be to have a system inside a bus to announce the next stop. Sounds like a weekend project right?

Think a bit further:

- Hardware needs to be resistant to harsh diesel engine vibrations

- 3G/4G connectivity

- Need a mobile data contract with local ISP

- Software needs to be able to handle network disconnections

- GPS needs to be able to pin-point at which stop the bus is currently stopped, even with bad GPS coverage in larger cities

- If the bus skips a stop because of a detour, the software should be able to detect it and announce the next stop

- The announcement should be bi-lingual for Airport busses

- The announcement should work for people with hearing aids

- Server needs to know all bus-stops

- Server needs to know different stop types, such as bus terminals, intersection stops, regular stops, hand-over stops, virtual stops

- Server needs to be able both work of a yearly bus schedule and real-time update

- Server needs to output in an understandable JSON/XML because the government subsidies demand an open-data format

- Server needs to publish data to an open-data server because of the subsidies

- Because the bus-company is sponsored by European Union money the control interface should be translated to German/French/English/Spanish

And now this weekend project takes a team of 10 engineers working for a year.

10y agoHN ↗

This exists in offline versions.

I think it just counts the number of stops and has some input from the driver

10y agoHN ↗

But, adding the wireless connections and the servers and the technical staff to code and maintain, would save the driver the labor of pressing the button. That's what technology is for.

10y agoHN ↗

It could just use the open the doors button with some hysteresis for opening/closing doors few times in a row in some short time.

10y agoHN ↗

KISS applies here. By keeping the 'driver presses a button' interaction, the complexity is reduced by at least a factor of 5, and the TCO ends up being much less.

Sometimes, a doorbell just needs to be a doorbell.

10y agoHN ↗

Sure but if I add all the above complications then I get automated bus tracking system where everyone on their phone can see where the bus is and when it's arriving, I can track and automatically detect issues with traffic, with the bus, with the driver.

Sure you can do the simple announce but the even easier one is to just have the driver talk over a mic, he needs one anyway. There I one upp'd you on KISS. But there is a reason not to KISS.

10y agoHN ↗

You can publish wireless / Internet data on where the bus is, but keep the driver pressing buttons as the source of the data.

10y agoHN ↗

That's how it works in our (German) city. It works great.

10y agoHN ↗

How often do you want the driver to press buttons? And how do you mark the intermediate positions in between bus stops? Because on any remotely complicated high congestion route, you will not get reasonable estimates if you only update at bus stops.

10y agoHN ↗

but the driver forgets to press the button and then people are affected. ah, was that my stop, why wasn't it announced?

10y agoHN ↗

That's why bus drivers get training. And if they don't perform as expected, it's not that hard to hire a better one.

10y agoHN ↗

I am so glad you said this. With the boom of the tech industry, it feels like a lot of stuff is getting over engineered these days. Instead of products that focus on something that does it really well, it does its job mostly alright with half-broken bells and whistles.

10y agoHN ↗

Except asking a bus driver to keep track of button pushing is distracting and could endanger the lives of the passengers or people outside. The stop needs to be announced 30-60 seconds before they arrive at the stop so then is driver is still actively driving. This is something the drivers union would be all over, it's not as straightforward as you assume.

10y agoHN ↗

The stop needs to be announced 30-60 seconds before they arrive at the stop

Why? You can announce the next stop a few seconds after departing from the current stop, and then again at the opening of the doors.

10y agoHN ↗

If there's a long gap between the stops, people may forget what's the next stop. Announcing at the opening may be too late or the stop has to be long enough. Even for slow passengers in a crowded bus. Announcing a minute before the stop is a good heads-up that bus is approaching the stop and people should prepare.

10y agoHN ↗

Then make the delay for the announcement of the next stop adjustable, configured according to the timetable.

10y agoHN ↗

A skilled engineer weighs the costs and benefits of a system before architecting it.

Is it more economical to pay for constant cellular connectivity across 1,000 buses, support IT staff for when the system fails or needs upgrades, etc. or to have the guy you're already paying $15/hr press a button after each stop?

10y agoHN ↗

At least on most UK buses, the driver already needs to tell the computer where on the route the bus is so that it can issue the correct tickets to customers. This has been the case for decades.

10y agoHN ↗

Or... microphone and speakers for the driver to announce the stop. If you'd like visual indicator too (accessibility FTW!) plenty of low tech and/or cheap solutions that would meet the requirements.

10y agoHN ↗

Or... microphone and speakers for the driver to announce the stop.

Breaks down quickly if you need bi- or tri-lingual announcements, which is the norm in many popular tourist destinations.

10y agoHN ↗

Laminated card in the cab with phonetic spellings for the other languages.

Aside: I'm impressed with the amount of bike-shedding for this one. Though I'm part of the problem, too :-)

10y agoHN ↗

Ha, indeed. Just thought of another - even lower tech - a printed list of all stops, visible to passengers on the bus, then clear signage for each stop. If the bus stops every stop by default, then this is simpler, if not then the passenger needs to be able to see this on the approach, with another time to signal that they want to get off. Can this get even lower tech?

10y agoHN ↗

Really? Either it's a location 'Piccadilly Circus' or its a place type 'Airport' - any of those can easily be learnt by the driver. Or writen on a printed sheet that they can easily see. Don't jump to making it more complex than it needs to be.

10y agoHN ↗

I see you have never tried it in practice. Train conductors in Austria/Czechia/Germany – which tend to be more trained and less busy than bus drivers – tend to stumble around for half a minute for each unintelligibly badly pronounced sentence. Even the pre-recorded messages in buses are of so bad quality that tourists prefer asking other passengers rather than trying to figure them out.

10y agoHN ↗

All London buses do this. I don't think you need a server, 3G/4G connectivity or even GPS really. You might be over thinking it.

10y agoHN ↗

On the other hand, the overthinking in question (which is probably true for bare-bones next stop indication) provides the base for my personal bus bug: know if/when a bus is reaching a stop from outside the bus in question, especially with geofencing, in order to know whether the bus I want to take is running early/late/not.

10y agoHN ↗

They actually do this as well (and many have live updating signs), so the buses do have GPS linked to a server of some kind. It's normally absolutely fine, but on the couple of occasions for me where it broke it seems that the server doesn't conduct the bus in any way and kind of hopes the bus is on the right route.

That's all speculation though.

10y agoHN ↗

They actually do this as well (and many have live updating signs)

I don't know about London, I've seen some systems where the stops had "live-updating signs" but only provided the time of the next official passage. And while I wasn't clear, my bug is not just stop signage but phone access as well (that's most useful when e.g. it's raining cats and dogs and you want to avoid waiting in the rain at uncovered bus stops)

10y agoHN ↗

There's a standard by Google for transit timings, and Translink, the public transport authority in my state, outputs these times. They're live.

Apps like 'Transit' also (accurately) show the current GPS location of the bus as it approaches, and it updated every minute (I believe)

10y agoHN ↗

OK, that's really cool. Definitely means they have "a server, 3G/4G connectivity [and] GPS" though.

10y agoHN ↗

Not necessarily. I think some of the systems use radio links to a central location and transponders at fixed points on the route to track the buses.

10y agoHN ↗

The TFL system will remove a bus if it doesn't get an update often enough that includes actual progress on the route. This is in fact the opposite problem of what you mention, in that they don't do a very good job of predicting based on past performance. E.g. I used to take a specific bus to pick my son up from nursery that always would "disappear" for a certain number of minutes at a certain part of the journey because a very short stretch of the route was always extremely congested at that time of day.

It's a tricky one, because they have big concerns about over-promising (e.g. the bus could be stuck behind an accident instead, in which case guessing that it's 5 minutes away because of past behaviour will really annoy people) so in some situations they err on the side of caution.

But they are unable to get rid of all of the corner cases. E.g. a bus that moves rapidly for a short stretch, then gets "stuck", will often appear, disappear, appear, disappear, with much less progress in time estimates than elapsed time should indicate.

The thing is, the team working on it has a huge amount of gathered info on how to give these predictions in a way that tries to minimise surprise and avoid making promises that cause problems for people. E.g. often it's better to take a bus off the list so people who are in a hurry will grab a taxi, rather than "promise" the bus will be there in just another few minutes. They still get it wrong too often, but I really don't envy them trying to get the right balance between promising too much and too little.

10y agoHN ↗

This is available for public buses in NYC. The city added GPS to their buses and then the ability to query distance from current stop via SMS.

Then they opened their APIs, and now I have a half dozen apps to choose from that will show the approximate time until the next bus from a particular line will reach any given stop.

(Not always 100% accurate, but useful enough to know when to wait indoors for 5 min if it's raining, or pick another line/choose to walk because of delays.)

10y agoHN ↗

know if/when a bus is reaching a stop from outside the bus in question, especially with geofencing, in order to know whether the bus I want to take is running early/late/not.

We had that, through GPS + GPRS (or SMS?), back in 2001-2 in a small city I was living in Europe. Complete with full (flash based) website map of where each bus is.

I used it to get out of the tech park/institute I worked at and into the bus stop outside at the very last minute (since bus timetables where completely unreliable).

10y agoHN ↗

GPS is insufficient to give accurate enough estimates these days. It may or may not work depend on your local routes and stops, but it's too imprecise for a lot of locations and traffic conditions to give decent precision. It's good as a "first approximation" and if you don't have anything better. Most systems today end up integrating data from multiple different sensors to improve on gps.

10y agoHN ↗

Sydney even uses the electronic ticketing system to provide an estimate on the capacity of the bus.

10y agoHN ↗

Thats clever. Most public bus systems I know in the US still see frequent cash payments onboard (the elderly, mostly). Is this not the case in Sydney, or is the system cashless?

10y agoHN ↗

Over the past few years they've rolled out a new electronic ticketing system which is used exclusively.

On some bus routes (and/or outside of peek times) you can pay a cash fare on the buss, but even that is still a single-use paper RFID card that should be tapped on and off. These sales are in the extreme minority though - the elderly require an electronic card to get their discount IIRC.

10y agoHN ↗

Really?! Where can I see this? On an app? I admittedly only use Google Maps when I'm in Sydney.

10y agoHN ↗

This is the only thing I want. Displaying a bus timetable can be done just fine with laminated paper, and I'm not convinced that the ability to update (with cancellations, route changes, etc) justifies a tech solution.

Displaying the actual bus status, though, is huge progress.

10y agoHN ↗

Halifax, NS, was doing that in the mid-'80s without the benefit of GPS or handy mobile apps. (Every bus stop/route had a phone extension, and major stops/terminals had talk boxes.) Granted, it wasn't a huge system, but it was hitting rocks with sticks and hollering over tin cans and string compared to what's available as a starting point today, and it worked. The idea that it should be a hard problem today, given GPS, vastly improved wireless communications and eleventy-seven different ways of figuring out where a user is (if entering a stop code seems like it might be too much work) is just plain silly. Or we've significantly redefined what "hard" means.

10y agoHN ↗

Or expectations have increased, both in terms of accuracy and in keeping costs low.

Consider that e.g. in London the old route indicators on the buses used odometers. Which worked for telling you were along the route a bus was. Most of the time. But didn't tell you how long it takes until the bus reaches the next stop other than on average, as it would tell you nothing about traffic etc.

Today London buses don't use just GPS to replace it either, because that's nowhere near good enough. See the "Tracking" section here [1].

https://en.wikipedia.org/wiki/IBus_(London)

10y agoHN ↗

When I said "worked", I meant worked, not approximated working or came lose enough not to have caused too many upsets. If expectations have grown to be higher than plus-ten-second accuracy (early wasn't tolerated), then we collectively need some sort of counselling/therapy.

10y agoHN ↗

You don't get that kind of accuracy in any system that has to deal with congested roads, or any dense major city. E.g. there's plenty of bus-stops near me where stops on the same route involves going around corners and traffic machines and stopping close enough that GPS would be insufficient to be guaranteed to tell the stops apart if you don't get a good enough fix on additional satellites quickly enough.

Even with all the extra data, and a lot number crunching, in London it's a challenge to get it to withing a few minutes during rush hour.

So, sure, if you're dealing with simple, small systems with little congestion, you can simplify.

10y agoHN ↗

German bus drivers have it as well. The routes are lists of stop names and their change is triggered by opening the doors. Can be adjusted by the driver if stops are passed, etc.

10y agoHN ↗

Runs on some windows if I remember correctly and routes are probably hardcoded along with the data for the segment display outside above the windscreen.

10y agoHN ↗

Many cities also have an app that displays how many minutes the bus is expected to run late, based on when it passed the previous stop.

10y agoHN ↗

what about the bus stop broadcasting a signal with the id, name etc? the system is activated by the proximity..so you can have smthing like approaching station A, leaving station A.

it is scalable because the bus-stop transmitters are added , moved, removed as needed.

updating the data can also be done remotely

10y agoHN ↗

That's more expensive because the cost (installation and servicing) scales with the number of bus stops.

Also, there are things you want to do that this doesn't cover. For example, announcing the next stop when you pull away from the last stop.

10y agoHN ↗

The announcing the next stop could be achieved with a dual radio setup, one high powered and low powered. The high powered could be used to detect an upcoming stop and the low powered could be used to detect proximity to the stop.

(Lets ignore the security issues that have been shown with amplifying the low powered radio in cars for a second) It's a method used in keyless entry in cars. A higher powered radio is used for button press on the cars key so you can unlock the car/open the boot/sound the horn to locate the car at a distance but keyless unlock should only work at a much closer distance.

Not saying its the correct way to do it (my local bus network doesn't do this) but stop proximity can be worked into their idea.

10y agoHN ↗

My local bus network use M2M Sim's running on the Vodafone network but its basically a virtual private mobile network running on top of Vodafone's network with no connection to the internet.

These connections are in both the busses and the stops. The reason is so the stops can provide real time updates to people at the stops too. If a bus breaks down its no longer tracked on the real time updates so you are not effing and blinding because a the stop said there is a bus due in 5 mins and no bus comes. And if a bus is removed from service unexpected then the information on why can be feed to people waiting for the bus. People are less likely to be pissed that a bus didn't come when expected if they are given a valid reason.

Personally I was hoping they used some form of radio link I could of sniffed with a SDR so I could put a live updating sign in my house. I just scrape the data from their live feed they use to update the tracker on their mobile app/website instead.

Broadcasting out a signal from the stops could need a transmitter that may need to be licensed as the signal would be to be picked up quite a distance from the stop. Also not all the stops in my area are these "real time updating" stops as they are normally put into covered stops (EDIT: Bus Shelters... The correct term wouldn't come to mind while I was typing this.) and some of our stops are just poles in the ground with a sign on them. The busses tracking their own position wouldn't need these stops upgrading to give this information to people on board the bus as the covered stops can be quite a distance between each other and removes any issues if a stop get damaged and stops broadcasting data (say a car crashes into it).

The busses tracking themselves also have benefits that don't visibly benefit passengers. The services back office can monitor busses and spot problems on the network as they happen. Unexpected heavy traffic on a route may trigger the company to skip or stagger a bus (one of my local routes in advertised as being "every 5-10 mins". The real time traffic condition information could also be sold on to other companies (not sure on the value of this data esp for busses that run once an hour or even longer).

10y agoHN ↗

I was surprised my local network didn't do something similar as it seemed like a logical choice. But after some research I found nothing about them using RDS to do this.

I would love to be able to talk to a tech at the travel network about this. My own research came to the suspicion that because where I am in the country they would of had to strike up deals with multiple transmission sites to get the coverage needed.

10y agoHN ↗

you need it. what if you want to connect to something and make some special announces.

10y agoHN ↗

What special announces? Either the bus driver does it or it's one that's used frequently (like 'Please move down inside the bus'), in which case there is a button for it.

10y agoHN ↗

Same in Munich. I think those work offline, also the driver can pick the next stop.

I guess it is easier in the long run to have all routes mapped and instead of GPS you use something like doors open / doors close as a trigger to jump to the next entry.

10y agoHN ↗

I work part time at the company that manages the new style signs for bus stations (EDIT: in London, to be clear), so we've gotten to see a bit of the size of the operation TFL has to deal with gathering and processing the bus information. He might have some details wrong, but he's certainly not gotten the complexity level wrong.

10y agoHN ↗

One of my "fights" at work is prototypes. A prototype can be build very quickly (cheaply) and can exhibit many problems that may be overseen by the "heavy design process". A prototype is a marvelous way to get feedback from users very early in the projects.

10y agoHN ↗

The only downside to prototypes is that then the managers don't understand why the actual project takes so much longer and end up asking to just go with the prototype for production.

10y agoHN ↗

Get better managers.

I've never had this problem.

10y agoHN ↗

The GPS and mobile data are overthinking it. The routes are known so simply trigger on door open/close (with some logic for denounce/etc) and cycle through.

For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved.

And the entire data set for the route should be able to live offline onboard the bus. Changing bus/numbers routes should include the step for updating route data.

10y agoHN ↗

Except that quickly becomes inaccurate if the bus driver decides to re-open the doors for an incoming passenger, or due to traffic conditions, opens the bus doors further from the stop than the beacon reaches. Or skips a stop due to road conditions.

10y agoHN ↗

1. Set a minimum time until next stop can be announced 2. Add a button to skip a stop

10y agoHN ↗

And then when the driver accidentally hits the button and needs to go back?

Notice how we're now adding more complexity now anyway though...

10y agoHN ↗

While active RFID is possible, other radio tech designed to work as beacons is more likely

10y agoHN ↗

Consider a bus stuck in traffic. It is not unusual, at least in London, for a driver to open/close the door multiple times between stops when traffic moves slowly enough, often with longer intervals in between than there usually would be between stops.

10y agoHN ↗

Set a minimum time until next stop can be announced

This feels like a dangerous hack that will come back to bite you in any number of unexpected scenarios that probably occur frequently during an average bus driver's day.

10y agoHN ↗

Bluetooth beacon (estimote?) at stops would help.

10y agoHN ↗

For an robust system I would use multiple data streams including bust status data/beacons/GPS. I've seen busses with offset stations.

3G is necessary for live ETA, if you want to get fancy. All these complications, in my view, are worth it if you want a system that works 24/7 and serves thousands of people daily. The HW and SW dev costs are acceptable, but it does need a decent team, depending on what the starting point is and the ability of the team members.

10y agoHN ↗

changing bus/numbers routes manually is not feasible. in complex networks this happens daily a hundred times.

10y agoHN ↗

I'm confused, are you suggesting there are places where bus routes change hundreds of times a day?

I can't imagine many reasons a route would have to change on short notice (accident blocking a road?), and in those situations the announcements not quite being right would not be the end of the world.

10y agoHN ↗

Buses are often assigned to a route and then leave that route unexpectedly in the middle of the day to service another route (accident, other bus broke down, traffic, subway shutdown etc) , often not starting at the beginning of that route.

10y agoHN ↗

Oh right, that makes more sense. I guess you could just keep a record of all the different routes in every bus though? And skip to the right spot manually as needed.

10y agoHN ↗

Then the bus changes its route number - certainly in the UK I see the drivers punching in the route details when they take over a bus.

10y agoHN ↗

I've worked in transit. This is not only feasible, it is how it is done.

10y agoHN ↗

cannot see why you would want to do it manually. it is done automatically in munchen.

10y agoHN ↗

For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved.

Yeah, and now we need to install and manage thousands of beacons in bus stops in pretty hospitable conditions.

Of course, GPS isnt perfect either. It tends to fail in CBD areas.

10y agoHN ↗

This has all kind of usability issues (now you're expecting the driver to pay attention to and correct mistakes), while providing less accurate data (buss pulls away from stop, gets stuck in traffic; estimates at next stop are based on the bus having pulled away and average times; customers gets angry) and less ability to do traffic management (e.g. spacing out buses by slowing down later buses slightly) without having drivers guessing and reporting data back.

For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved.

That's not remotely accurate for the reason above.

And the entire data set for the route should be able to live offline onboard the bus. Changing bus/numbers routes should include the step for updating route data.

It can. E.g. the old London system used odometers. But that was sufficient only for showing "next stop ..." and for reporting very rough estimates. Today it uses odometers, gps, rate gyros and turn rate sensors to be able to more accurately position the buses along a route, and they still regularly "give up" and take a bus off the schedule when traffic conditions means the uncertainty is too high.

10y agoHN ↗

And now this weekend project takes a team of 10 engineers working for a year.

More like 2 guys working for a month.

10y agoHN ↗

I'd say ~ 5 Engineers working for ~ 3.5 months

(I calculated the geometric mean of the numbers both of you have given, in case you're wondering ;) )

10y agoHN ↗

Generallly my approach if engineers disagree significantly on a timeline is to multiply the worst case estimate by a large factor, on the assumption that odds are good it means neither one has a clue what the actual complexity is if they don't manage to get to rough agreement...

10y agoHN ↗

Absolutely impossible. There is hardware involved.

I'd say something like 3 people for 6 months to 1 year (not all 3 being needed 100% of the time).

10y agoHN ↗

You've just described a cellphone running apps that already exist.

10y agoHN ↗

Far more useful is the system which tells you, at the bus stop, what the ETA of the next bus(es) is. And that really does need the whole GPS+data system, along with physically robust signs. But on the other hand, that can still be done by a small company - I know there's one here in Edinburgh but I can neither remember their name nor find them in a search.

You can do quite a lot with a small local company. It's global reach that's expensive, as soon as you start needing any kind of high-touch sales.

10y agoHN ↗

I don't think anyone would have assumed that you can build such a system in a weekend.

Half of your points assume that current transportation companies don't have any digital data about bus stops and reroutes.

Anyway I haven't been in a bus that doesn't have such a system for at least for a few years (in western Europe and China).

10y agoHN ↗

Half of your points assume that current transportation companies don't have any digital data about bus stops and reroutes.

The question is what is more work: Using the existing data that is stored in some obscure format for which you first have to write an importer/converter or just recreating the data. Add the fact that the existing data might need some cleanup/additions (e.g. times on timetable are only stored with minute precision and you want second precision for more accuracy).

10y agoHN ↗

Alternatively, you could have the driver press a button.

10y agoHN ↗

This seems like a problem that has been solved many times in the Western world.

10y agoHN ↗

You can do this with a RPi and a RFID reader. Every station would have a tag, when bus stops tag is read and your script just announces the station.

10y agoHN ↗

You generally want to make the announcement before the bus stops so that people can be ready to get off when the bus stops. Also buses don't always stop at every station on it's route so have to deal with what happens when stop is skipped.

10y agoHN ↗

You can certainly do it with an RPi - although you might have to keep rebooting it every hour or so. :)

GPS and 3G/4G are trivial add-ons. There are stacks for mobile data.

RFID doesn't have the reliable range, so that won't work as a solution.

It's not really a hard problem for a competent small engineering team with a mix of embedded hardware and software skills. It's not quite a trivial problem, because there are some interesting edge cases, mostly when a bus is diverted. But there is absolutely zero rocket science required.

You could hack something unreliable together in a weekend, but it would take up to six months to get all the bugs out, do the production engineering for a bullet-proof solution, standardise the installation process (if retrofitted), and so on.

Double that if management is poor. :)

Those last parts often get forgotten in software. Hacking something together that kind of works if you don't look at it with a visible frown is hobby coding, not engineering. Engineering is the boring work that happens after that, to build something with a quantifiable many-9s uptime that meets or exceeds a standard spec with comprehensive test cases.

10y agoHN ↗

Usually the announcement is not "we have stopped at Foo and Main". The announcement is "Next stop Foo and Main" while you're on the way to it, so you have time to sling your bag back over your shoulder, get your coat back on, and make your way towards the door. Depending on the density of stops along the route this can come anywhere from 1-7 blocks ahead of the stop.

Source: I don't drive,and ride the bus a lot.

10y agoHN ↗

But nobody wants that. They want conductors back. This is not a problem with a technological solution.

10y agoHN ↗

They want conductors back

That's just faster horses though.

Conductors aren't a solution to a problem in themselves, they're a means to solve a problem. So what's the actual problem? WHY do they want conductors back?

10y agoHN ↗

Move to China, every bus in Shenzhen has a lady conductor.

10y agoHN ↗

- Hardware needs to be resistant to harsh diesel engine vibrations

That's a non-issue.

- 3G/4G connectivity, need a mobile data contract with local ISP

This is a duh!

- The announcement should be bi-lingual for Airport busses

It can always not be. How would that be worse than NOT having the announcement at all? Besides what would the other language be? French, German? That would still be useless to 90% of foreign visitors...

10y agoHN ↗

Besides what would the other language be? French, German?

I guess the names of the stops are untranslatable anyway :)

10y agoHN ↗

Canada. Here it would need to be bilingual by law.

10y agoHN ↗

umm I travel in GO and DRT everyday. Neither have bilingual announcement. Same with Mi-way.

10y agoHN ↗

Check out APRS[0]. Solves most of the above already by centralizing data-collection. Would just need a beacon on every vehicle. No need for internet connectivity. Receiving stations can be scaled up as required. Logging of course not an issue as can be logged on-board and/or externally.

The on-board display can overlay a map and call the next stop based on proximity. Not a weekend project, I'll admit, but many existing moving parts available off the shelf.

Hams play with this stuff... a lot.

[0] https://en.wikipedia.org/wiki/Automatic_Packet_Reporting_Sys...

10y agoHN ↗

You are massively overthinking the problem. The very first system of this type I saw didnt even have a microcontroller. Just an eprom for driving LED array, bunch of TTL logic and two inputs. One linked with 'open doors' switch, the other going to undo switch. And it worked fine for couple of years in a >million city.

10y agoHN ↗

Complicated, but not that bad... just need to optimize for the worst case and the best case will purr along happily.

Some sort of near-field beacon on the stops might be more durable than relying on GPS, but it won't help with missed stops.

10y agoHN ↗

Every variation of this comment/mindset drives me nuts. It's unbelievably thoughtless.

10y agoHN ↗

The question is why do someone needs the first 50Mi primes in a C++ file

10y agoHN ↗

No idea, but it would be a fairly reasonable way to test a prime-finding algorithm - or to replace a prime-finding algorithm with something less complicated.

10y agoHN ↗

Because the alternative: generating the primes at compilation time with a template functional expression would be too slow, and quite possibly would explode the compiler's memory limits. (No Turing Machine here), and doing it at run-time or deliverying it as a "resource file" is obviously out of the question.

10y agoHN ↗

I find I've had to bar myself from thinking about many of these factors, after I realised how many projects I stop myself from pursuing because I know about the hard bits, and realise how many of my past projects I'd never have taken a shot at if I knew what was coming up and was thinking like that.

It's great to know, but often it's not so great if it leads to inaction...

E.g. if you have a "I could do that in a weekend" impulse, sometimes it's worth following that impulse. If you're right, you've spent a weekend building something potentially great. If you're wrong, you've spent a weekend learning about things you clearly didn't know in advance, and the lessons can often be very different from the objections you'd have throw up if you thought about it without doing, and sometimes what you learn may provide the impulses for another new idea.

But it's hard. E.g. my first impulse for anything that requires collecting money is how painful it is to deal with credit card processing and reporting and VAT and accounting, because I've done big complex billing systems with heavy reporting requirements, and compliance complexity etc.. It's very hard to put that impulse aside, and "just do it" and think about how to solve the payments afterwards. But that's how I did things he first few times I needed to handle payments.

10y agoHN ↗

Yeah right, except that you can't. It takes so much time to nurture something and build it in a way that satisfies the user.

10y agoHN ↗

The sad truth is that in the majority of cases the quality or time taken over the implementation isn't particularly important other factors matter much more:

* building something people actually want. * advertising * engaging with users * aggressively promoting your product. * customer service

It doesn't matter if what you did was hard or not. It matters that you can acquire and retain customers.

10y agoHN ↗

That $12M/yr figure for indexing 1B documents seems way off to me.

10y agoHN ↗

Yes and if you look at the footnote where the arithmetic is laid out, it seems quite confused. It conflates querying with indexing and also flatly assumes that each machine will index 5M pages, which seems low to me.

10y agoHN ↗

Too high. See http://news.appbase.io/scaling-elasticsearch-writes/ for example:

In doing these tests, we indexed over 1 billion documents under just two hours. The peak merge rate we saw was 129 MB/s, yes our ElasticSearch cluster on a 24 node setup is gobbling data at this speed.

Most mind boggling of all: At our peak sustained ingestion rate, we would have added over 13 billion documents in a single day and this would have cost us a mere USD 244.

Their test documents are much smaller (100 bytes) than a typical web page but if you multiply that figure by 50 to get to 5kb documents (probably a realistic average web page size minus all markup and assets), you get 12200$ for 13 billion documents or 1220$ for 1.3 billion documents (assuming that scales linearly). I'm probably way off in my estimate myself but I'm convinced that a few thousand dollars is much closer than the proposed 12M$.

10y agoHN ↗

Yeah, but asking readers to resize windows, resize the font, and tweak the CSS to get decent line heights is a bit of cheek when it's trivially easy to implement at least minimally pleasant-to-read typography and layout.

It's a bit shortsighted of a writer to completely ignore the realities of readability. The current layout may conform to some vague idea of purity or minimalism, but that puts form over function in a big way.

10y agoHN ↗

Resizing a browser window is hardly as involved as tweaking CSS or browser settings.

10y agoHN ↗

Au contraire, it's a lot more readable than your link, because it respects my window size.

10y agoHN ↗

It's readable for some subset of people, but on a 32" monitor it's worthless - I tried to read it, but just abandoned it in frustration.

10y agoHN ↗

To me this post addresses the same naivety that David Graeber demonstrates when he claims that everyone not involved in hammering out a sheet of steel to make a horseshoe or pulling a plough is engaged in a bullshit job[1].

I think it's not a coincidence, therefore that many of the people I run into who advocate a universal basic income, arguing tooth and nail against a job guarantee (the YC guys no different) is young, tech savvy and overly optimistic about the scope of automation to subsume all human labour within months.

[1] http://strikemag.org/bullshit-jobs/

10y agoHN ↗

Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.' I don't think he thinks "anyone not involved in hammering steel" is unnecessary, that's a bit of a strawman. The argument goes that big corporations have a tendency to hire more people than necessary because middle managers are rewarded based on how many people work under them. Or that people looking to keep their jobs, try to make themselves look more busy and valuable than they really are.

People waste a lot of time at offices doing things that aren't work. But our culture demands we be at work 8 hours a day, even if the work only requires 4 hours. Economists in the 1930's thought the world would be so rich by today, that people would only need to work a few hours a week. And the economy has grown vastly since then, and we are much richer. But I don't think most people have the option of working that little, even if they want to.

But that's a different subject entirely from automation. Those jobs can really be valuable and still be vulnerable to automation. AI has improved a lot in the last 5 years. I don't think it's implausible at all to imagine most jobs done now being partially or totally automated.

10y agoHN ↗

Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.'

I don't think that's really true, especially in a big corporation. Transparency of the value pipeline in such institutions isn't great at any level, and big corps (and similar large institution -- government often has the same problem, for instance) have a tendency toward not making even the concept of the value stream that justifies the position known to the people working in it much of the time. Its quite likely that the people working a position don't understand how its supposed to deliver value (and are in a poor position to see the big picture of whether it does), while the people who do know how it is supposed to deliver value don't see the small picture of what actually goes on (and thus are in a poor position to evaluate whether it does deliver the value it is supposed to.)

There's no particular reason to think the people working in the position are in the best position to evaluate value delivery (or that even the people in the best position, whoever they might be, are better at answering the question than a Magic 8-ball.)

10y agoHN ↗

There is a great incentive to obfuscate when your "value delivery" is skimming (mutual funds that charge 1.25% management fees for underperforming the S&P), privatization fraud (health insurance in the US, government contractors, and pretty much everything related to real estate since "home appreciation" is now supposed to pay for people's retirements instead of pensions), taking advantage of stupid people (lottery, penny auctions, telemarketing, almost all advertising), education fraud (textbook industry, "student loans now, regret later" private colleges and universities - which since many student loans are federally guaranteed is also a form of privatization fraud), or "intellectual property" racketeering.

10y agoHN ↗

I think that a lot of this is because things that seem quite pointless on an individual level are actually very important at scale. When I started at Google my job was pushing pixels around on the search results page, and when I left my job was leading a team pushing the pixels around (in the middle I got to do actual cool stuff). On an individual level, this is pointless and boring. On a corporate level, some of those pixels were worth literally $1B+. I would love to have a billion dollar business, but I can't get there from scratch by pushing pixels around on a page.

10y agoHN ↗

Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.'

Yeah I know that's his point and that's why I describe his point of view as being naive.

It may be true that for some percentage of workers at any given time their role is experimental and may turn out to be unimportant or a failed attempt at change, and it may be the case that it take a while for these sorts of inefficiencies to be rooted out or resolved because the cost and time associated with doing so make it a low priority but that is something that people who are in those positions would not necessarily be aware of.

This is the same as when people say things like "I don't know why we are so precious with our kids these days. I never wore a bike helmet and I survived!".

Without the proper perspective people frequently form wildly inaccurate opinions.

I don't think he thinks "anyone not involved in hammering steel" is unnecessary, that's a bit of a strawman.

If all I have to do in order to make it a straw man is replace the word iPhone with horseshoe he's doing a pretty good job of constructing his own straw man.

The argument goes that big corporations have a tendency to hire more people than necessary because middle managers are rewarded based on how many people work under them. Or that people looking to keep their jobs, try to make themselves look more busy and valuable than they really are.

That story is completely ridiculous.

People waste a lot of time at offices doing things that aren't work. But our culture demands we be at work 8 hours a day, even if the work only requires 4 hours.

Then businesses have an opportunity to compete for talent by offering better working conditions. If they can do so profitably then they win.

Economists in the 1930's thought the world would be so rich by today, that people would only need to work a few hours a week. And the economy has grown vastly since then, and we are much richer. But I don't think most people have the option of working that little, even if they want to.

This is more a result of the spoils of productivity increases not being distributed evenly.

By having massive unemployment and under employment then expecting the remaining workers to work harder for less pay on average while more and more income is paid towards servicing private debt the finance, banking and real estate sectors have made out like bandits.

But that's a different subject entirely from automation.

I don't think so. It has a lot to do with automation because that is where the productivity increases have come from, but rather than the federal government guaranteeing full employment and setting a floor price for labour and a minimum set of conditions below which we consider employment to be exploitation, which would not only maintain demand in the economy but also reduce the inequality with which the spoils of those productivity increases are distributed, they have left people to fend for themselves and the economy has languished as a result.

Those jobs can really be valuable and still be vulnerable to automation. AI has improved a lot in the last 5 years. I don't think it's implausible at all to imagine most jobs done now being partially or totally automated.

Maybe in 1,000 years but I disagree that automation will have the type of impact people (particularly in the UBI advocacy camp) claim it will.

10y agoHN ↗

Weird example, where it's assumed that a single machine can only hold 50GB of document data, which results in the assumption that you'd need 200k machines to store 1T documents...

10y agoHN ↗

I feel if you have something better to offer, say a better way to look at information, not just a better page rank algo, but a different way to look at information that is scattered across, one that enables a better insight into existing web pages and less overhead of constant 'searching' by typing a query, something that automatically helps to "see" relevant information with less effort, then it can be an alternative to what a Google search does today and can position itself as a competitor in the long run even if it does not exist today.

10y agoHN ↗

I have worked with one-man companies that did this and often they are shoddy products. Healthcare just seems to suck at software.

Full-stack developers are already a very fast evolution from the front/back end developers, which were an evolution from a guy doing just HTML/CSS in the 90s. Now you have to include sales, marketing, negotiation, project management, and business management skills on top of your full-stack development skills. The result is that something is going to suffer unless you are a genius, the product is super-simple, or you have 20-30 years to have learned all of this.

There's a much more interesting question in the form of: "Why's that company so big? A tight-knit team could do that in a weekend".

10y agoHN ↗

Apps aren't just a pile of features, they're also ecosystems.

In the early years of Twitter, it would have been relatively easy to clone Twitter, but you won't have had the ecosystem.

Poster example: Google+

10y agoHN ↗

Besides underestimating the scope of the problem, one logical fallacy that programmers fall for is believing that a single developer should be able to juggle lots of tasks easily. If a small team can create a SaaS application, why do large companies need so many engineers for so many specific things?

I think, at a certain scale, it becomes necessary for organisations to ensure that there are certain people keeping specific things running. Let's say, a company doesn't have any specific person assigned for server administration and then one day, is confronted with a DDoS attack. Who is supposed to handle it? Whose responsibility is it? You have few people who know the basics of operating servers but no one who can swiftly respond to it. If no clear role is established, then the company is bound to face more attacks and possibly, lose more customers.

10y agoHN ↗

Features and scale are huge things.

Maybe you could make, say, a Twitter clone, in a weekend. But you couldn't make one that can handle anything near Twitter's scale nor that is as feature-rich as the actual thing, and besides, nobody will be using it.

10y agoHN ↗

Everything seems easier when you're not doing it yourself. It's only a minor variation on the grass being greener. Everything gets way more complicated when you're doing it at scale, with high reliability, for real users, with some hope of making money. Practically nobody adds complexity for its own sake. Usually, if you think something is much more complicated than it needs to be, it's because you don't have any idea what the real requirements and constraints are.

10y agoHN ↗

The article raises some good points but there is still a dichotomy here to me. Compare a successful startup with 2 founders with an established company with 2000 employees in the same business. The scale of the difference in employees vs the difference in the product is often shocking. BigCo product might be better, but that much better?

Obviously the quality of the product won't scale linearly with the number of engineers, but still, startups punch above their weight. Maybe it just highlights how much harder things get when the company reaches a certain size.

10y agoHN ↗

Those 1998 extra staff don't have to make the product 1000x or even 2x better (however you might quantify that). They just have to make it sufficiently better to maintain or gain market share.

10y agoHN ↗

I often hear this logic from inexperienced engineers. My answer to them is: Then why don't you do it? <grin>

Recently a junior engineer said this to me about Uber, and I helped him to understand - His logic was "it's just a taxi app".

There were loads of Taxi apps before Uber and loads of them after Uber. Yet none of them were uber?

Sure that a medium complexity CRUD app can do the pickup and scheduling, and spit out some json to the app.

But what if you have 100,000 journeys taking place simultaneously, and you must track their positions so that you can render the journey map on the invoice?

Can you do live traffic planning and re-route around the busy traffic on the fly?

Does your weekend include the full billing system? This is a mandatory requirement for a business.

Can you spot busy periods and increase the fares according to the laws of supply and demand? All online in realtime? Can you continuously A/B test this to optimise profits?

Can you deal with the fact that maps and roads are always changing?

All of these and more are the things that make Uber, Uber, otherwise you're just one of those shitty taxi apps that failed.

Still think you can build it in a weekend?

10y agoHN ↗

And even that is only a small part of what makes Uber, Uber. If you want to actually compete with Uber, then once you've gotten all those minor details worked out, you're going to have to tackle the real problem which is dealing with 100's of different regulatory frameworks for offering taxi services, being a taxi driver (or a not-taxi driver as the case may be), hiring of private contractors and two dozen or so other entirely non-technical things neither you nor I have considered.

10y agoHN ↗

That's what the problem is. You can invent something in a weekend; the red tape takes longer. I think that still counts as "built in a weekend."

10y agoHN ↗

Your comment makes no sense. Invent is a pretty meaningless word in this case. Invent doesn't mean 'make a product capable in working in the real world where it has to follow laws, physical limitations, and deal with stupid people'.

If you invent an app in a few hours, then get fined out of existence your app sucks.

If you invent an app in a few hours, then it crashes under high load, your app sucks.

If you invent an app in a few hours and your code sucks and you can't adapt it later, your app sucks.

10y agoHN ↗

If you invent an app in a few hours, then it crashes under high load, your app sucks.

Yes

If you invent an app in a few hours, then get fined out of existence your app sucks.

No

10y agoHN ↗

Apps that stop working because the company folds definitely suck.

10y agoHN ↗

> If you invent an app in a few hours, then get fined out of existence your app sucks.

No

Sorry, but back here in reality apps must comply to regulations. It's your job as a voter to make sure your regulations comply with reality.

10y agoHN ↗

For one, not everyone cares about the rules. For two, rules rarely apply to all people all the time. For three, this is external to the apps and you can hardly judge the app for what arbitrary regulations their location has at that time.

10y agoHN ↗

Caring or not caring about the rules is irrelevant if those rules make it impossible for you to operate.

Uber has famously skirted rules and has gotten away with it largely because they've been able to throw VC money at PR, lobbying, and paying fines. Unless you happen to be similarly lucky funding-wise, your first round or two of fines for breaking rules can easily shut you down permanently.

Maybe it's a bit hyperbolic to say that this would mean "your app sucks", but the end result is that you built something that no one can use and you can't make money of of. That sucks.

10y agoHN ↗

> dealing with 100's of different regulatory frameworks

Isn't ignoring regulations the most often quoted reason for Uber's success?

10y agoHN ↗

Having an effective strategy to skirt regulations probably requires just as much knowledge of the regs as a company who complies with them.

10y agoHN ↗

I don't think that's really true in this case. In the few cities I know any details for, the Uber answer was, "we'll just do it until the city sues us". That doesn't really take any special knowledge, just preparedness with the lawyers.

10y agoHN ↗

Having an effective strategy to skirt regulations probably requires just as much knowledge of the regs as a company who complies with them.

Not really. The "effective strategy to skirt regulations" seems to be "when someone makes an issue out of it, we'll burn investor money on PR and lobbying against the regulations, and on paying any penalties that get assessed for violations".

For that, you don't need much advance study of the regulations, just a deep supply of investor funds.

10y agoHN ↗

Ignoring regulations is easy.

Getting regulations to ignore you is the hard part.

10y agoHN ↗

They don't "ignore" regulations. They violate regulations. They do so strategically or they would have been shut down long ago.

For example, if public opinion weren't on their side, perhaps due to, say, Uber rides being as dangerous as they're sometimes claimed to be rather than as safe as statistics show them to be, some of those "little" city lawsuits could have blown up much worse.

10y agoHN ↗

Unfairly, IMHO. It certainly helped, but they did so much, and better, than the other apps and they made legit, value-improving changes to the ride-hailing process: actually acting on feedback, dynamically setting prices, coordinating a movement away from tipping, applying modern IT.

Remember, their core product was not violating regulations: it was filling empty spots for legit black limo drivers, and was more expensive than taxis but still competitive because of the quality of service.

10y agoHN ↗

That and the economic recession where lots of people who had vehicles were now trying to make ends meet. Also, they paid their drivers better initially until they build up their driver network. Of course leveraging the media helped and a symphony of other things all pushed them to where they are now.

10y agoHN ↗

Something I find really interesting is that Uber drivers in the US are ordinary people. In the UK, they are straight up taxi drivers. It's the same brand for two totally different implementations of the idea.

10y agoHN ↗

You couldn't.

No way you can delegate work to 2k engineers and tell them what to do for next 2 days. Physically impossible.

10y agoHN ↗

Yes, Amdahl's law.

I'd probably just use the 10 most productive engineers, and tell the others to prepare for testing :)

10y agoHN ↗

I don't want to sound bad, but again you couldn't do it.

Say you have 2k engineers for next two days. Where do you begin?

10y agoHN ↗

Describe the app generally, and leave it to them to form their own teams and solutions. Hackthon style, and hackathons at least usually have functional prototypes by their end.

10y agoHN ↗

You are correct, but are taking these guys too seriously. They know about the mythical man month and are just riffing, for better or worse.

10y agoHN ↗

Not my problem - I'm their manager. If I have 2000 engineers I just make one of them Lead Dev and then shout at him when he hasn't been able to deliver after I gave him 1999 software engineers. What an idiot.

10y agoHN ↗

Nope. Nine women can't make a baby in one month.

10y agoHN ↗

Also, it's often the case that it's better to buy a tool and integrate it into the company than build from scratch. It's especially true with a small team. It's not the most romantic idea and sometimes engineers feel that they could build the solution themselves in a weekend.

10y agoHN ↗

I'm pretty sure uber doesn't do any of that locations or traffic routing stuff you mentioned. When I take uber the driver just has their favorite app if Waze, Google or Apple Maps for the actual routing or directions.

10y agoHN ↗

It has to for the time estimates and finding the 'closest' driver to you.

10y agoHN ↗

For me the time estimates are always super incorrect and seem only based on the distance.

10y agoHN ↗

I use to live next to a highway with the closets exit being a few miles away and I would constantly get drivers on the highway as my closest driver. I always had to cancel since they were actually ~15 minutes away, not 3-5.

10y agoHN ↗

I have found that Uber's estimates get better with time. When Uber was newer in my city, it used to give ridiculous underestimates. They've become far more realistic now.

10y agoHN ↗

Or the passenger in the case of Uber Pool. I couldn't believe what Uber expected of a driver when a new Pool request came up. They had to hit a button, look at the screen, rethink how they were going to go somewhere. Oh and then worry if the requester will cancel.

That happened on my last trip. I felt bad for the driver so I helped him out finding the best way to pick up a passenger after the first two cancelled.

I think Pool is a good thing but not in rush hour traffic where going slightly off the route adds 10 minutes to a trip minimum.

10y agoHN ↗

I'm pretty sure Uber uses Google Maps in for a lot of it's mapping and navigation [1]. I'm not saying Uber adds no value - payment processing isn't easy, but it's not solving any new mapping problems that Google hasn't already solved. Its main innovation has been to ignore existing taxi laws by classifying itself as "ride-sharing".

[1] http://www.zdnet.com/article/uber-to-invest-500m-in-mapping-...

10y agoHN ↗

I doubt Uber would have gotten started without leaning on Google for both mapping and directions. This benefited them both as Google incorporated Uber back into it's Maps product.

10y agoHN ↗

Uber initially (and still does) rely Google Maps for a lot, but they have an internal team working on their own mapping, and have spent a lot of money acquiring map-related tech.

10y agoHN ↗

Now. But they initially bootstrapped it with Google Maps, which was sensible to do for their initial MVP, which is all an "I could do that in a weekend" project is.

10y agoHN ↗

Can you provide a source for the claim that Uber violates taxi laws?

As far as I'm aware, Uber doesn't violate taxi laws. Uber started out providing "black car" services which are not classified or regulated as taxis. Never have been, still aren't. Black car services are ones where you schedule a trip in advance, and there's no "hailing on the street". Being able to hail taxis is what makes them taxis and subject to regulation.

A lot of the drivers who first began working for Uber were existing black car drivers, and who operated their own black car business on the side. Many still do. Uber didn't come along and make this existing class of ride service illegal.

The only vehicles subject to taxi regulation are ones that offer rides to people on the street without prior arrangement, as far as I'm aware.

10y agoHN ↗

I agree with everything you say, but that response, like the junior engineer, focusses on the technology. So give people six months not a weekend -- there's a good chance they still won't be Uber.

The success of many companies, and probably all of the unicorns, has nothing to do with technology. The tech is necessary, of course, but so are desks and an accounting department.

Internalizing that has been difficult for me as an engineer.

10y agoHN ↗

I know - but I was talking to a junior engineer, and I was concentrating on the technical aspects to inspire him.

I expected him to go off and actually attempt building it, that way he would get some more experience.

And he did! He went off that weekend and wrote a proof of concept route planner, it was great to see him learn something new and complex

10y agoHN ↗

many a junior engineer who persist on the idea its easy, can be done quicker, cheaper, and such, also rarely have the needed dedication to see it through especially when it takes longer than expected, missing requirements are identified, and such.

I have seen multi million dollar projects hit the same snags, horribly under scoped all because Mr Easy didn't think about asking all of was needed. They saw the tree and missed the forest

10y agoHN ↗

Proper project scoping (or to rephrase: identifying the true core business needs) is truly one of the hardest problems in tech. Over-scoping makes you end up with a ponderous system that doesn't serve core business needs very well, while underscoping can lock you into poor initial decisions that cripple your ability to expand into other related functionality.

What's the expression? "Plan your system for 100x growth, but build your system for 10x growth". Something like that except for functionality rather than scalability.

That and supporting the system. Junior engineers tend to think about building the tech, which is honestly the easy part. Keeping it working until the last of your customers are finally ready to upgrade 10 years later is harder than junior engineers tend to think.

10y agoHN ↗

Ha!even this sounds like a mistake to me. How about "Make the simplest thing work for one person.". Make sure they get some value, and then iterate and repeat. Once you've clearly solved the problem at hand then start to scale, optimise etc That is of course all after you've presold the solution.

I know it's not always easy to do this, but that really is part of the challenge. There's still too many deathmarch projects lead by people trying to do things 'the right way'. If you're working on a new project for the love of science, please make sure you're releasing something valuable every single week if not every one or two days!

10y agoHN ↗

Sometimes it is the tech... just not in obvious ways. You have to look beyond the basic spec.

Uber and Lyft are not just taxi apps. They are massively polished. When I use them I am astounded at how good they've gotten the UX and then I shudder at how hard that must have been across two mobile platforms and in the presence of very unreliable networks. I've ordered an Uber with no cell Internet using barely usable coffee shop wifi from a block away.

Then you have scaling, which brings up or creates issues you just do not see at smaller scales. Big O type considerations and weird edge cases that only come up once every billion transactions start mattering.

Then there is all the big data stuff. These services do tons of route optimization and analytics.

So yeah marketing, sales, fund raising, management, etc. also matter but tech often matters as much or more. It's just that junior engineers usually only see the tops of icebergs. This is why despite decades of pure biz people saying "tech doesn't matter" you still rarely see founding teams succeed without at least some tech founders but you do occasionally see even lone tech founders make it. (But those tech founders do have to know or be willing to learn the non tech stuff!)

10y agoHN ↗

Internalizing that has been difficult for me as an engineer.

Technology exists largely to accelerate the pace of development and of course, it's never a sole driver of any business. There is no reason why you can't create a small-scale uber with call centres and excel spreadsheets but I doubt it would reach the scale and impact of what Uber is today. Granted that, it's hundred times easier to figure out to technology than business, I won't trivialise role of engineering because it's something allows a business to near its global maximum.

10y agoHN ↗

It has a lot to do with the technology. It's not the only or even the most important factor, but it's still one of the most important.

I think a big part of the disconnect being discussed originates in the myth that CRUD-based apps are fundamentally easy. This simply isn't true. The challenges of building a modern, complex, scalable CRUD app with good UX and a maintainable code base are not 'less than' challenges rooted in algorithms or bit bashing. Doing it well enough and fast enough to be competitive in the market is HARD.

10y agoHN ↗

The success of many companies, and probably all of the unicorns, has nothing to do with technology.

This is 100% true and unfortunately gets distorted from the financial backers who pump them up for unicorn status. Quite a dichotomy:

"Software is eating the world!"

"The hard thing isn’t setting up an organizational chart. The hard thing is getting people to communicate within the organization that you just designed."

10y agoHN ↗

The success of many companies, and probably all of the unicorns, has nothing to do with technology.

So why not look at failure then?

10y agoHN ↗

Its harder to learn from failures than to learn from successes.

Success has the false-positive issue. Failures have the false-negative issue.

Cargo-culting false-positives is going to be less harmful than cargo-culting false-negatives. (X Company didn't use stack-ranking and then failed. Clearly stack-ranking is bad for companies)

10y agoHN ↗

Maybe. Just to muddy things even more, some successes are failures.

Have you ever met a billionaire who said, "I was lucky" and left it at that?

10y agoHN ↗

Yeah. At least with a success you have a data point that it could be successful - with a failure, you don't even know that.

10y agoHN ↗

It's not the technology success/failure axis in question, but how much weight to give it relative to other factors.

AKA in predicting overall success would a team with low tech expertise, but high sales expertise be have a better chance?

10y agoHN ↗

Of course software is eating the world. Wheels ate the world. Fire ate the world. Written language ate the world. Plumbing ate the world. Electricity ate the world. Radio communications ate the world.

Not every project or human endeavor that involved wheels, fire, written language, plumbing, electricity, or radio ate the world, but every project that did not involve them eventually got eaten, for the simple reason that you needed to keep up with technology as a minimum. But beyond that, projects succeed or fail for the same reason they always have: understanding the actual problem you're trying to solve.

Uber's success is not due to software. Uber's success is due to its understanding of the transportation market and how to play the regulatory game to its advantage while never getting permanently in trouble. Uber's ability to deliver on the huge market opportunity they saw—i.e., its lack of failure—is due to software. We quite reasonably call this "success" when talking about technology, but if we're talking about companies succeeding or failing, that's not a discussion about technology.

10y agoHN ↗

The success of many companies, and probably all of the unicorns, has nothing to do with technology

Have you considered broadening your personal definition of technology past computing and engineering?

From Wikipedia:

Technology can be the knowledge of techniques, processes, etc. or it can be embedded in machines, computers, devices and factories, which can be operated by individuals without detailed knowledge of the workings of such things.

Technology is the creation of systems. Those systems run as much on the users of that technology, and the members of the organisation supporting it, as they do inside of microprocessors.

10y agoHN ↗

The success of many companies, and probably all of the unicorns, has nothing to do with technology.

Most of the unicorns are founded with engineering talent coming straight from Stanford, MIT, Harvard, UCB, Waterloo etc. you name it.

Making a company successful is as much about economics of scale than raw engineering and of course, both are closely related in this day and age. Engineering shapes what and how your products feels like in real-time (= understand very short code-to-production lifecycle).

It might not be politically correct to say but I am not here to make you feel better. Anyone who has actually been in a team at a company that has exponentially grown will tell you the same: your chances to ever survive that "first wave" of users is directly proportional to your ability to hire the best, which of course is a lot easier when they are your current or former classmates.

That's not to say there aren't companies founded by people not going to the schools mentioned above or that engineering talent is only available at these places. It is obviously not the case.

However, the density of talent at top-schools is absurdly high. And this is a game changer for most early stage companies and even more so for soon-to-be-unicorns.

10y agoHN ↗

Probably why a lot of professors say that University is often about networking opportunities just as much as it is about learning.

But aside from knowing talented people, it takes visionaries to generate fantastic outcomes. Walt Disney for example stopped drawing cartoons himself early on because he had better artists around him. However, he had the vision that ultimately brought the best out of each artist.

10y agoHN ↗

The success of many companies, and probably all of the unicorns, has nothing to do with technology.

Execs love to think this but it's just not true. Top heavy companies waste money. It's true that tech of not the only ingredient but for a tech company it's critical. Good tech is not a commodity. Top-tier engineers are worth as much as a good CEO. That is if you want to remain lean.

10y agoHN ↗

At a previous job, we had serious development bandwidth issues. The company was trying to solve that by hiring more developers, and all it was doing was making things worse.

The root cause of the development bandwidth problem was that we wasted vast amounts of time and energy on building misfeatures. And we built so many misfeatures because we had way too much development capacity in relation to our product planning capacity. So the BAs didn't have enough time to fully develop ideas before sending them to engineering, and were instead stuck throwing things over the wall when they were still half baked. All these half-baked ideas took time to build, and then they took time to fix after they bombed on the market, and they slowed everything else down since every new feature is something you have to ensure still works when you make a change in that part of the software. Day in, day out, we were taking on technical debt to cover our design debt bills.

So then we tried Agile, and man oh man did that make things worse. 'Cuz now instead of just lobbing things over the wall, the BAs had to be actively involved in every sprint. There goes any opportunity to sit down and concentrate, because now you've got a nattering horde of 20 developers all trying to get clarification on last week's decisions, which leaves less time to make this week's decisions, which means next sprint things will be even more sketchy and ill-defined.

The correct solution would have been to hire more business analysts, and they key thing we needed from those business analysts wasn't instructions on what to build. It was instructions on what _not_ to build.

10y agoHN ↗

The company was trying to solve that by hiring more developers, and all it was doing was making things worse.

You don't mess with Brook's law!

10y agoHN ↗

I know I've seen this pattern in person, but something about the way you wrote it makes me want to go back and think about some of my other projects before I could identify that was the problem.

I know one project in particular we did a release about every 7 months. Every release cycle they estimated 2 weeks for requirements, every cycle took a month, and yet the next cycle you'd see a 2 week estimate on the roadmap.

As for switching to Agile, I know some people would tell you that Kanban would identify this problem for you by around week 8. Did you guys ever try having development liaisons help them draft requirements that were precise from an engineering perspective? Sometimes that initial meeting works better when there are only three people in the room, instead of a bunch of exasperated people being condescending to you again while you're trying to do a good job. adjusts collar nervously I'm uh... asking for a friend.

10y agoHN ↗

"...precise from an engineering perspective?", this (in my experience[0]) doesn't work. Because requirements are messy and always change. As they say, " No plan survives contact with the enemy." and "No requirement survives contact with implementation." The reality is that you need to come up with a timeline that accounts for this problem (say 8 weeks) and only scope the release to fill 50% (4 weeks). An 8 week release cycle may even be on the long end of a release cycle.

Why 50% of a release's entire timeline? Because you're going to have overruns, errors, re-dos, and any number of random unplanned elements creep into the scope of your release. And the reality (I may get blasted for this blasphemy) scope creep is fine. As long as it pertains to the originally scoped requirements and is clearly adding value. "A delayed game is eventually good, but a rushed game is forever bad.", Shigeru Miyamoto. The important part of that quote is "delayed", you still need to make sure you release otherwise you're just spinning and wasting company time and money.

And whatever you do, don't try and "fill in weeks". If the release is ahead of schedule, let it be. Let it finish weeks early if it has to. Then, rinse and repeat. Admittedly this type of release process can be hard to sell because there's usually a negative emotional response to the idea of only scoping 50% of the total allowed release cycle and that somehow the other 50% is waste. It isn't, it's there to guarantee that the items scoped initially actually get done on time. That way QA can get to them in a reasonable amount of time, give feedback, and (during the second half of the release) engineering can fix the issues raised by QA. Business development can also rest comfortably knowing that the features scoped into the first half of the release are actually going to come out on time.

I know it sounds like "You're just adding slack." But it's intended to be more than that. If you see teams attempting to squeeze more items into the 50% release you know there's a prioritization problem within those teams. Which can be remedied by either: 1) reducing the overall size of the release cycle further (eg 8 to 4 weeks with 2 weeks as the 50% mark) to provide more flexibility into what goes into the product in a shorter time frame. 2) Helping those teams clarify what the high-level product direction and priorities should be.

Hopefully this 15 minute rambling comment is valuable to you. If not, I apologize.

[0]The mistake I made and the advice I can personally give, don't focus on up front high-accuracy estimates and designs. Come up with a reasonable design, your best guess estimate, and then implement as furiously as possible (coding best practices should still be enforced). You'll know much earlier in the release cycle what the real timeline for your feature will be than if you'd tried to spend all that time up front.

10y agoHN ↗

I'm not sure you agree with the parent if the parent is saying "no, it's not possible, because the tech is so hard", and you're saying, "it's not an issue of technology, but the business side".

10y agoHN ↗

Internalizing that has been difficult for me as an engineer.

I suspect this is because everyone wants to not only feel as part of something but they want to make an important contribution. I venture to guess accountants at Google do not feel they make a difference to search. The culmination of parts is what makes the company whole and all those parts are required.

Its hard to accept you are not as important as you wish you were. Its a great step in the maturation process of one's professional career though.

10y agoHN ↗

The tech is necessary, of course, but so are desks and an accounting department.

Internalizing that has been difficult for me as an engineer.

After I got over that particular hurdle, I would still think:

Ok, so "build it and they will come" is not a thing, got it. But still, engineering is most important by far. Without sales, support, marketing, or legal, you still have a product, just one that is adopted slower. Without engineering you don't even have a product to sell in the first place.

And that's true. But in the end it's just splitting hairs. It doesn't matter. Over the longer term, no successful, scalable business is ever going to exist solely as an engineering team. Every job function has a purpose[0] and gets you where you want to go faster; in some cases, lack of certain non-engineering job functions will make it impossible to even do certain things.

So sure, you can bootstrap a product with just a few engineers and nothing else, but unless you're ok with it being a niche-market product that you spend all your waking hours working on, an engineering-only org is never going to cut it.

[0] With the obvious exception of those people hired to satisfy the latest management fad du jour, or to give the CEO's deadbeat nephew something to do with his time.

10y agoHN ↗

I thought like those junior devs for a while, too, and I agree to your sentiment.

The feeling that the CEO I was working with was doing mundane tasks was astonishingly - he even said that he could to his whole job using his smartphone.

But after trying to do the business stuff myself I realized that no one in all of this process is irreplaceable - finding a good software developer may not be easy, but it's also not easy to find a good CEO. This is important to realize that software developers shouldn't see themselves as some type of elite.

The main problem is that junior devs don't realize that 90% of business is based on networking and being friends with people who are in middle management and want to increase their reputation in their big corp. You don't get big clients because you're a genius hacker, most of the time those B2B applications are simple CRUD apps. You won't get that $500k because you simply didn't play golf with the key partners. I'm not even exaggerating, I've seen that the best deals are made in ones "spare time".

Maybe startups are another breed than the software companies I know, but don't drink the cool aid and don't assume that technology is your main differentiator. It's working with people all along.

I think it isn't even about scaling high-performance systems, it even starts with the mindset of those who think that software developers are entitled in some way, usually junior devs and people who never did any real business work.

10y agoHN ↗

To me the question isn't "Think you can build it in a weekend?"

You mentioned 5 product "concerns" that Uber must contend with to make a point. No doubt if we brainstormed more we could come up with perhaps 10 or 20 more such "concerns".

Now let's be very generous and assign 10 engineers to each concern.

I still come up with a maximum of 200 engineers for Uber.

But Uber has 2000 engineers.

I don't see any creative ideating that would justify that many engineers _as necessary for the top or bottom line operation of Uber_.

They are there for other reasons but not because Uber needs that many engineers to operate.

10y agoHN ↗

To repeat myself from an earlier comment. How does one even arrive at the number 200 based on the complexity of the problem? Secondly not all 2000 will be working on their core app. There will be solutions required at each level. For example in some countries cab services are provided by larger contractors employing drivers and not individual drivers themselves. So Uber needs to have a billing system which collates data from their individual drivers and then calculates the payout to the company. This requires a separate set of engineers, not working on the core app. And then there are HR systems, Finance Systems, every bit they are not contracting to an Indian Service company has to be handled by an "engineer".

10y agoHN ↗

Okay. Then how many engineers would Uber need to have for you to say "that's too many"?

10y agoHN ↗

I don't really know. What I do know is people seem to misunderstand engineering team size as a simple sum of people working on the Uber app. It is not only the android/ios developer or the full stack developer who is an engineer in Uber. An IT administrator who helps people, technical or non-technical, with their work laptops also qualifies as an engineering team staff to the outside world.

10y agoHN ↗

What makes it your or my place to decide that? They're a private company, and they have plenty of profit motive for efficiency. If they didn't think they were getting value from all of those engineers, then they would fire them and save themselves some money. Or are we going to argue that corporate executives aren't greedy enough?

10y agoHN ↗

You mentioned 5 product "concerns" that Uber must contend with to make a point. No doubt if we brainstormed more we could come up with perhaps 10 or 20 more such "concerns".

Now let's be very generous and assign 10 engineers to each concern.

I still come up with a maximum of 200 engineers for Uber. But Uber has 2000 engineers

There's two ways to interpret this: 1) Uber is overstaffed by an order of magnitude. 2) You are underestimating the complexity of Uber by an order of magnitude.

Uber might be overstaffed, but I suspect the answer is mostly that you are drastically underestimating and don't see the complexities of running a large-scale service in multiple countries, dealing with regulations, worldwide scalability, time and cost estimates, surge pricing, tracking and mapping, payroll, billing, payments, apps, internal IT, etc., etc., etc. There aren't 10-20 "concerns". There are hundreds.

I bet they have at a hundred engineers just working on billing and they're all overworked.

I don't see any creative ideating that would justify that many engineers

Because most of Uber's engineers aren't there to build "creative" solutions. They're there to enable the business.

10y agoHN ↗

Is it not a problem to people that this many people would be required just for billing? I don't want to grossly oversimplify but at some point people need to start getting judgemental and saying, "no, we have created something too complex. We have to simplify."

10y agoHN ↗

You clearly have never worked anywhere close to payments :)

Kidding aside, the regulations around anything that handles money are quite complex, and the liability issues mean that you'll spend a lot of time on getting it right, too.

In theory, payment is super-easy. In practice, any payments solution that is not just a toy eats engineers for breakfast. Stripe is probably one of the leanest ones - and they have 400+ engineers, lean on existing infrastructure, and cover only 25 countries.

10y agoHN ↗

I agree with your two interpretations. But to clarify that is my position: I believe Uber is overstaffed by an order of magnitude.

It seems after reading through this thread that I am definitely in the minority on that opinion, but I'm not sure if I understand the position there.

How many engineers would Uber need to have before one would say "okay, that's too many". 3000? 5000?

Uber can afford 2000 engineers. That doesn't mean they are anywhere near good at utilizing them, which seems to be the implicit assumption in many comments on this thread.

Google has in the tens of thousands of engineers, which they can afford to have because of Search. I highly doubt they are anywhere near reasonably utilizing those engineers to top line or bottom line product value.

10y agoHN ↗

Netflix is another example. A couple of thousand programmers and tens of thousands of VMs to build a top drawer e-commerce site. I built a drawer-below-top e-commerce site with twelve people in a year, and a third of them were muppets.

I interviewed at another well-capitalised streaming video place, and they also seemed overstaffed. They had a permission service that worked out what people could watch, based on purchases, subscriptions, and bundles. Something that's basically one SQL query. They had something like six people working on it full time.

10y agoHN ↗

How many engineers would Uber need to have before one would say "okay, that's too many". 3000? 5000?

However many it takes before the cost of an engineer is higher than the value added. I have no idea where that line is for Uber. Maybe it's 10K. Maybe it was back when they hit 50.

Uber can afford 2000 engineers. That doesn't mean they are anywhere near good at utilizing them, which seems to be the implicit assumption in many comments on this thread.

This might be true but it doesn't support your point. If Uber is bad at utilizing engineers, that doesn't mean they should employ fewer. In all likelihood, they'd be about as bad at utilizing their workforce if they cut it in half.

10y agoHN ↗

> I still come up with a maximum of 200 engineers for Uber. But Uber has 2000 engineers

There's two ways to interpret this: 1) Uber is overstaffed by an order of magnitude. 2) You are underestimating the complexity of Uber by an order of magnitude.

The third interpretation is that we have no idea what else Uber is working on besides the operations that are visible to us as consumers. Uber can afford to pay 100s of engineers to build out a few technical concepts/prototypes that will never see the light of day. They basically need to do this, because their unicorn valuation isn't based on chasing the vanishing margins of the taxi industry, its based on owning an entire market that won't exist until they create it.

I suspect its a mix of all three.

10y agoHN ↗

Exactly, Uber needs to scale along axes in a way most traditional web companies do not.

The axes have been pointed out are heterogeneous city/state/nation regulations, labor law, naviagation/connectivity issues, etc. And onto each of these diverse cases, there needs to be a way to analyze results consistently and deploy business objectives from a centralized corporate strategy.

As you point out: to justify its unicorn status, Uber needs to prove that it can succeed not just in competing with taxis in the major markets where it already exists, but that it's technology platform can be deployed to the edge cases, and the areas where it hasn't.

So what the extra engineers are doing finding a way to acheive user reliability under the reality of accelerating changes to requirements and sharding of policies for separate operational units that will result from forced-growth into non-optimal markets.

10y agoHN ↗

I think the critic has the benefit of hindsight which may translate to 1000s or hours saved in DEV. E.g. Copying Uber verbatim versus all the experimentation to get there.

Hence it's not uncommon to see clones of things (e.g. Dropbox, slack, etc) developed at fractional cost.

Your junior may be onto something , get shares in his. Ew startup quick !

10y agoHN ↗

I was offered to interview at a company, they wanted me to do a coding "pre-screening" which consisted of creating a full-stack, and fully operational registration website from scratch.

Maybe I'm a super slow architect and developer, but I'd estimate that to be a greater than 8 hour time commitment.

I assume that the org's leadership acted in good faith, that someone honestly think's a non-slapdash solution would be a couple hours of work. It boggles my mind.

And yes, I declined their invitation to pre-screen and thus did not complete the interview process.

10y agoHN ↗

Just to confirm, you meant just register -> store to database + forgot password and validation right? (Greater than 8 hour time commitment do sound about right, but I'm just curious to make sure I read it correctly).

10y agoHN ↗

The instructions did not give an upper bound in terms of functional requirements, though the lower bound of "realistic use" was given!

So yes, I would assume that includes things like full email validation round-trip too.

Even just slopping together an in-memory, no-email-sent deployed solution is at least 3 or 4 hours in my book, which seems way excessive for a pre-screening question!

10y agoHN ↗

And you had to do that from scratch? No libraries allowed? That does seem rough. With libraries allowed you could probably spin something up with rails + scaffolding + devise though.

10y agoHN ↗

maybe libraries are allowed, but I don't know any that can really turn this into a 30 minute project. And even researching libraries is hours of work :) If you have any suggestions for this on nodejs, I'll be happy to add it to my interviewing toolbelt :D

10y agoHN ↗

Cookie cutter Django. 30 mins seems on the high side.

10y agoHN ↗

Sorry for the late reply, Just noticed your comment. I don't know a way to do it in node. If you have setup devise a few times recently you could possibly get it done with ROR + the devise gem + scaffolding. This is a pretty terrible interview question though, as testing if someone has practiced a canned solution ( which is really all you could do in that time) doesn't seem that great, and actually taking the time to make your own solution would take much longer and an 8 hour unpaid interview question also seems bad </rambling>.

10y agoHN ↗

What about user editable profiles with the associated valudations on email change ;)

10y agoHN ↗

Depends which framework you pick for the solution. Using Django what you propose is doable within an hour.

10y agoHN ↗

a nodejs solution was the requirement, but maybe if the org's leadership was comming from a Python/php background they might assume using a scafolding was a given. I don't know anything like django for nodejs though!

10y agoHN ↗

isn't express.js kind of like a django for node?

10y agoHN ↗

No, Express is much closer to Flask than Django. One of my biggest gripes with Node is that there's really no equivalent to Django/Rails in terms of developer productivity and best practices.

10y agoHN ↗

Meteor is not at all equivalent to Django. I've used both, and they're leagues apart. Meteor still can't even use SQL.

If you choose Meteor, it's going to dictate your entire stack from database to frontend framework. That's fine if what you really want is an all-in-one solution but it's definitely not a Django equivalent.

10y agoHN ↗

That's what gives Express its appeal. Rather than a walled in environment, I can pick and choose the pieces that I need. Much more flexible if you want to move over to a different library within Express.

10y agoHN ↗

Express + Passport can easily be done in less than an hour. It's super simple to setup. I've got my default template for every site I whip up. Now writing that from scratch is a completely different story. I doubt any of their current engineers could just put that together in a day.

10y agoHN ↗

"If you guys were the inventors of Facebook, you'd have invented Facebook." The Social Network.

10y agoHN ↗

Fully agreed.

However, remember l that Uber (probably) started without any of those complexities. Probably just on a weekend.

So I think the absolutely incorrect "I can build this in a weekend" could be rephrased as "An absolute MVP (Or a proof of concept) like this product could be built in a weekend".

This would make sense, right?

10y agoHN ↗

"An absolute MVP (Or a proof of concept) like this product could be built in a weekend".

Maybe, theoretically, in some cases. But even so that's very different from the original "I could build that..." bloviation.

10y agoHN ↗

Yet none of them were uber?

You underestimate the amount of luck involved in becoming Uber. It's not all luck, but luck is a major factor.

10y agoHN ↗

Consider this then: If you could build an app as polished as Uber's, there are hundreds of minicab companies in the UK alone, most of which have apps massively inferior to Uber yet has to compete with them, and that are thus a market for better apps.

There probably is a lot of luck involved in becoming Uber, but if you can make an app that good in a short amount of time, then the opportunity cost in doing that and sending a demo round to a few dozen minicab companies is very, very low.

Yet these companies are still stuck with horrible apps, because in reality it takes a lot of time to develop a good one, even if a prototype is fairly trivial.

10y agoHN ↗

Data federation is the key to complex data. In a month or two, the first cut anyway, I doubt it could be done in a week-end.

10y agoHN ↗

Uber didn't start with all of that either.

The MVP can be built in a weekend though.

10y agoHN ↗

And then you have to get a million people to say, "I'm just going to Uber there." Except replace Uber with the name of your new app.

10y agoHN ↗

The hardest parts of starting and running a business generally have nothing to do with tech. That is even true for most software businesses. This is what most developers who have not started or ran a business don't truly understand.

10y agoHN ↗

Also, all the edge cases which become visible only after real users start pounding the service exposing all "this should never happen" conditions.

10y agoHN ↗

Can you spot busy periods and increase the fares according to the laws of supply and demand?

And beyond the technical issues: Can you effectively deal with the negative PR that results when you increase fares during a natural disaster or terrorist attack?

Uber isn't just an app, it's also a company and a brand and a service. It's something people understand, it's something that's popular, it's something that people rely on.

So sure, let's say a really good dev could crank out a software engine that does everything Uber does in a month. Now what?

10y agoHN ↗

You seem to be implying that really good devs are just lying around not being used whereas great PR guys and execs are the real value. I don't buy it.

10y agoHN ↗

The evidence of PR being effective is right there if you think about it.

10y agoHN ↗

The evidence of good tech being important is at your finger tips.

10y agoHN ↗

I'm not really sure how you got that from the words I wrote. There's tons of work in addition to the tech which leads to a success story like Uber. That doesn't imply that the tech doesn't matter, just that other stuff does also.

10y agoHN ↗

While I agree 100%, I think the answer is more satisfying when viewed a different perspective.

It's not that Uber needs all those engineers. It's just a question of marginal cost vs marginal return. Uber made 1.5 billion in revenue in 2015. An engineer costs roughly $150k / yr. An additional 100 engineers only have to make enough small improvements to increase revenue by 1% to pay for themselves.

10y agoHN ↗

An engineer costs roughly $150k / yr

This number is far too low. Uber is headquartered in San Francisco.

10y agoHN ↗

A lot of the work is convincing people:

1. Convince customers to download the app, give their credit card, and then trust their rides and selves to a driver

2. Convince drivers to clean up their cars, download the driver app, and turn it on, and wait for customers to ask them for rides

10y agoHN ↗

As an inexperienced engineer I did the same thing. My punishment for my hubris was to have bizdevs who felt the same way about us.

The weird thing is though, I still actively encourage new devs to make this mistake in certain circumstances. For one thing, telling them no isn't going to help, we have no shared context, and the math of their productivity versus mine works against me spending too much time convincing them.

If the problem has a logistical or bureaucratic component, once in a while the stars align and they manage to pull off something the department has been trying to do for years but always runs into a wall. Some people in an organization will assume something is true if they hear it from enough independent parties. I know if three people ask me the same question about a block of our code, I start looking at that code suspiciously. Clearly something is wrong with it. Sending the 5th guy to ask some IT guy or product manager he same question, especially in the way a new person tends to formulate questions, sometimes knocks the cobwebs off and the person decides to change their behavior or volunteer to do something they have resisted doing before.

A week of junior dev time for a 5% chance to clear out a bag of tech debt is a good investment IMO. And if it doesn't work they learn something about why things are the way they are.

I had a senior coworker who used to laugh and say "You're new here aren't you?" and I started giving him grief for doing it. Sometimes the new guy is the only one who can get things changed.

10y agoHN ↗

Great insight there.

Additionally, the new guy will be much happier when he's treated the way you suggest. I know I would have appreciated that instead of the bs I got at my first proper job in the beginning.

10y agoHN ↗

I always try to point out to the junior folks when a thing is janky and broken for "historical" reasons rather than technical reasons, in the hope that their fresh eyes on the problem will find a better solution. Sometimes we're still just coasting on the best solution we could come up with 2 years ago, and in those cases it's likely that we could do better if we re-think the problem from scratch.

10y agoHN ↗

I think you can ask a simpler question: how do you get traction, drivers and passengers included? You application will be invisible, nobody will see it.

10y agoHN ↗

And my answer to your question is: your reply is pretty far off base. Quite simply: when Uber first launch, Uber was not the Uber you are describing. Telling an engineer they need all these things from the start is completely wrong.

Yes, competing with Uber now, with "this can be done in a weekend", is naive. But that's not the point.

The point is, that there are plenty of problems you can make a lot of money on, with a timeline anywhere from a week to a year. Some will be sustainable, and some will be simple cash grabs. The hardest part is identifying not just what, but when. If you have enough connections in the industry, it can be much easier to predict this than if you are on the outside of things.

I have built plenty of things people would say "I could do it in a weekend". And yes, a weekend is a little short, but many of them took just a week, at least initially. The answer is not "Uber does all these things, you need to too." The answer is, "you need to predict the next Uber, so you don't have to do all the things Uber currently does."

10y agoHN ↗

ALso, they are literally trying to solve a problem similar to the travelling salesman problem when you consider Uber Pool scheduling, Uber Eats, etc.

10y agoHN ↗

True, but they don't have to find the optimal solution, or even anything close to it. They just have to find a solution that's slightly better than the next-best solution, which in most cases is very far from optimal.

10y agoHN ↗

I think a related thing people say (that makes me cringe) is that xyz is "just a CRUD app", quite dismissively. You could argue that the vast majority of applications are "just CRUD apps"! Once you get up to a certain scale, even "basic CRUD apps" are complicated.

10y agoHN ↗

You couldn't build the Uber of today without significant resources, however you could indeed build the version of Uber that made them popular and got them access to those resources in a weekend. Uber started out as a black car app. They got lucky, went viral, and the founders were well connected enough to get funding to build out much of the advanced functionality you're referring to after the app already became popular.

It isn't the advanced functionality and scalability you're talking about here (that was built in reaction to its popularity) that made it popular. It was their MVP.

10y agoHN ↗

Even more than that: code is only a tiny part of a tech business, even one that has code as its most critical foundation.

A classic example is banking or money transfers (like paypal). So many people think that financial services are nothing more than moving bits around, which seems like an easy thing to code. Aside from the fact that it's not as easy to code as one might imagine, that part is maybe only 1-5% of the business. Paypal got where they are not because of the quality of their code or their operations but because of their relationships. They established relationships with banks, with regulatory agencies, with governments, and they spent years and years doing these things. They signed contracts, they established trust, they created systems for interfacing with many different banks and governments (not just tech systems, but policies and personnel). The result is that paypal is now one of the easiest ways to move money around the world with minimal friction. Cloning paypal's tech wouldn't enable you to clone their success, for those reasons.

Same thing with Amazon. Amazon's user facing software is only a fraction of their operations, their main competitive advantages are selection, price, fast fulfillment, and customer service.

10y agoHN ↗

You're going to talk about Uber? Watch this video: [1]. He talks about every regret regarding their infrastructure. At first you kind of nod your head along with "yeah, it's not that easy", but as he keeps talking you can't help but wonder why the company has absolutely no internal technical direction whatsoever.

They employ over 2000 engineers (just devs, excluding support staff, etc.). They use micro-services designed to the point where teams can't reuse pre-existing services. They are rewriting the same functionality over and over, and over and over again, as separate teams don't even have a clue what has already been written. Pick a piece of functionality you need. There are straight up 20 different APIs that do the exact same thing, but good luck figuring out which one you should be using - oh wait, why bother when you can just write the 21st version of the same thing. They are at the point where they have no clue what their repositories contain. In the video, he can't even provide an accurate number of deployed micro-services. THEY CANNOT EVEN ASSESS WHAT THEY ARE RUNNING IN PRODUCTION. They're literally drowning, and cannot recover.

The point of the article is wrong. It's not about what can be done in one weekend, it's what can be done in 12-24 months by a competent team. Yes, Uber operates internationally which comes with a very crazy set of challenges. 2000 engineers worth of challenge? Not even remotely close. You should need fewer than 100 people who really know what they're doing, and another 300-400 people who can follow the lead of the first 100. If you need more than that for a single product (ie: Google does not apply as they have 100+ full products), you are doing things wrong.

If you're going to provide the anti-thesis, don't use Uber as your example. They don't even understand how their business is managing to remain afloat from a technical perspective.

[1] https://www.youtube.com/watch?v=kb-m2fasdDY

10y agoHN ↗

And on top of all of that, can you get 100K people to use your app?!

10y agoHN ↗

I think you're making a related mistake, which is to overestimate the importance of the tech to the business.

When Uber first released, my guess is they probably couldn't do most of what you list above. But it didn't matter, because they had one compelling feature: they routed around regulation. This allowed them to pay drivers more and charge riders less than traditional taxis, while taking a cut for themselves. This is a clearly profitable position to be in, which means that they have the money to bring in capable business analysts and programmers, and the rest of what you say above will happen naturally.

The taxi apps before Uber didn't recognize that the killer feature was routing around regulation, and the taxi apps after that didn't do it first.

Sure, I can't write Uber in its current state in a weekend, but I could absolutely write Uber's minimum viable product in a few days. The problem is having the right combination of a good idea, the correct understanding of why it's a good idea, and the expertise to execute it (not just technical expertise).

10y agoHN ↗

I think you chose a bad example. Imho Uber has waaay to many engineers for what they are doing: to the point where they need to invent work for themselves with no business value.

10y agoHN ↗

In France, the law (Art. R. 4228-10 du Code du travail) imposes one WC and one urinal per 20 male workers, and two WC per 20 female workers. So basically, it expects that up to 1 worker per each 10 workers will be busy with non-job related stuff.

Another datapoint, I heard that an accountant counted that the actual number was 1 for 50.

In any case, when you're alone or a small startup, this doesn't seem much, but when you have 100,000 employees, this means that between 2,000 and 10,000 equivalent employees are all the time in the WC (and being paid for that, basically).

Of course, there are a lot of other overheard that become very significant when you have a big corporation, but that still exist (and that we easily discount) for small startup and individuals.

10y agoHN ↗

In the US, most companies aren't hiring janitors, they outsource it to a facilities management firm or it's handled by building management.

I'm also not sure how that's relevant.

10y agoHN ↗

Ever thought that maybe that 1:20 or 1:10 isn't about handling steady load but about handling pre-/post-lunch peak loads without half hour lines?

(And I've seen the half hour lines, in an office that had two WCs for 100 people. THEN you end up with people spending 10% of their day doing something other than work.)

10y agoHN ↗

Do you stop thinking on the loo? I don't. For knowledge workers, that time isn't completely lost...

10y agoHN ↗

I searched for GTGACCTTGGGCAAGTTACTTAACCTCTCTGTGCCTCAGTTTCCTCATCTGTAAAATGGGGATAATA and found another post by the same author on another website http://bitfunnel.org/strangeloop/

I've seen some other posts by Dan before but didn't know what kind of things he works on because I generally don't check that because if I did then I'd end up just checking people instead of reading what people were writing. Anyhow, from the article linked above -- for which a video is available at https://www.youtube.com/watch?v=80LKF2qph6I -- it turns out that Dan has done quite a bit of work with implementing search related technology, and I think that was interesting. His contributions to BitFunnel can be seen at https://github.com/BitFunnel/BitFunnel/graphs/contributors

10y agoHN ↗

I thought you had accidentally pasted some of your genetic code from your clipboard. future internet problems

10y agoHN ↗

Anyone find the 528 MB C++ file that is referred to in the Twitter screenshot near the end of the article? (With only 57 lines of code but, oh, an array with the first 50M primes hardcoded.) What's that about?

10y agoHN ↗

that "probably" makes me wonder why you would say that? is there something about 50 million primes in there?

10y agoHN ↗

I don't think you really need that many primes for Project Euler, but I can certainly imagine someone tired of including a call to her prime sieve in tons of problems, and of having to waiting for it to run, and thinking, "let's just make a library that has all the primes I could possibly need in a constant array".

10y agoHN ↗

Actually it actually wrong to assume somebody could solve a trivial problem in a weekend, however some companies still have way too many people working at it. Consider twitter with their 3860 people. they could probably do it with half of them. especially the technical people are 40% of the company... way too much. i don't think that they need no people but well...

10y agoHN ↗

Yeah, this post seems to be addressing an annoying conversational experience, but ignores the other side: the reality, which you describe. I understand that just scaling out their platform is a ton of work that justifies having a lot of engineers. But how does Twitter have so many engineers on staff and yet their users constantly complain that they don't improve even seemingly low hanging fruit? Is ~1,500 people not enough? Is there some lingering technical debt they're dealing with? Or is it just people accumulated over time they value too much to let go?

10y agoHN ↗

Otherwise, indeed, it's a basic evaluation error, as described as long ago as in the The Mythical Man-Month; it just seems that nowadays, the multiplier is not 9, but even bigger.

https://archive.org/details/mythicalmanmonth00fred Figure 1.1, page 5

To go from a program to a programming system (interfaces, system integration), you need to multiply the effort by 3.

To go from a program to a programming product (generalization, testing, documentation, maintainance), you need to multiply the effort by 3.

So to get a programming system product, you need to multiply the effort from the program step, by 9.

Nowadays, we need to compose indeed multipliers for distribution, reliability, availability, localization, targetting multiple UI platforms, etc.

And the original multipliers are also probably too small, when you see the effort required just to integrate testing and CI on some projects.

10y agoHN ↗

"I could do that in a weekend"

You could but you didn't so I can't use it...

10y agoHN ↗

There's a naïveté in saying "I could do that in a weekend" as a measure for employee bloat, but that's making a straw man of the question.

It is nearly always more probable that a large company with several hundred or more plus engineers doesn't need that many. That those engineers are there for other reasons -- political reasons or speculating on new products, valuation reasons, etc. -- than for the reason that those engineers are fundamental to the top or bottom line of the company's core product/s.

Uber does not need 2000 engineers. That's a prima facie fact.

I've worked inside several large companies and saw this first hand. Hundreds of engineers across teams doing "stuff", no doubt, but nothing that ultimately or ever tied to the top or bottom line. The only ideas that I could come up with for why these people were there were: perhaps the headcount helped company valuation, perhaps we needed that many benchwarmers for when there was company churn, maybe there is some political reasons a manager wants a sizeable headcount on their team, or maybe it's just leadership ignorance as to how things are getting delivered. But in no way could I see a concrete justification for many teams and many engineers.

What am I missing?

10y agoHN ↗

Maybe some of those engineers really were superfluous. Maybe some were contributing to the bottom line in ways you don't understand or appreciate. I've seen plenty of engineers take infrastructure for granted, discount the value of research ("speculating on new products" is important), or even dismiss support as a cost with no revenue upside. Most often, as those people gain experience they also gain a broader view, allowing them to distinguish between complementary roles and truly superfluous personnel.

10y agoHN ↗

If you want to build a "boat" you start with a canoe, you don't build a ferry.

10y agoHN ↗

Businesses should keep adding engineers to work on optimization until the cost of adding an engineering equals the revenue gain plus the cost savings at the margin. This is often many more engineers than people realize.

I think this is the key realization, and it's actually a variation on Conway's Law[0]. In a nutshell, some part of BigCorp discovers (or believes) that by hiring a software developer for $X, they can make $Y > $X in return. So they do.

As long as some part of BigCorp is able to do that, you get more software engineers. This continues until you either (a) run out of budget, or (b) run out of opportunities.

Because it's not done "top down", it less efficient than it could be to get all of the features in total. But due to how these engineers are funded internally, it's not possible to do the "top down" approach at all. And that's okay, all that really matters is that $Y > $X.

[0] https://en.wikipedia.org/wiki/Conway%27s_law "Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations." In this case, it's not the communication structure that causes the perceived bloat, it's the funding mechanism. The basic idea that software organizations reflect some non-software, human structure in the business, holds.

10y agoHN ↗

Building an app and building a business aren't the same.

Go out and sell the app to people and find out how easy it is.

10y agoHN ↗

Fog Creek is something like 50 employees right? That seems like a reasonably sized team for their product, but Twitter/Uber are in a different ballpark.

10y agoHN ↗

googles and ubers of the world are not just their end-user facing apps + the backend required to support them.

these are large businesses with end user-facing "IT" side of the house being just the tip of the iceberg.

10y agoHN ↗

I was on the other side of this with a product that I was one of the main people on back in the late '90s - I looked at what we were selling and how much we were selling it for, and thought "Why are people buying this? Anyone with a decent IT department should be able to do this in-house."

But really when you sat down and looked at it, what we were selling was a supported and tested product that could be configured and active in less time, with excellent ROI, and at significantly less than the cost of dedicating an in-house developer to do it. What we were selling wasn't cutting-edge technology, it was time and cost savings for a department that was almost certainly already short of both money and staff time.

10y agoHN ↗

Generally it is Marketing..

And also "Hofstadter's law"

10y agoHN ↗

This is such a straw man conversation. Of course it is a fallacy to think "I could build that in weekend" and translate that to a claim about the inefficiency of a company. I think that's what is being responded to in this thread. But that's not the interesting debate.

A better consideration (imho) is this:

   For a given business/offering/product, what is a likely optimal number of engineers needed to continuously deliver the product's value efficiently? Then: How far is a given company from that optimal number?

This is a hard analysis to do. It is virtually impossible to quantify or put this kind of evaluation in a laboratory. There are too many counterfactuals that would be at play.

We really just have to consult our experience and consider which of these is the more probable:

(1) A business/offering/product team with lots of money is really good at utilizing its hundreds of staff. OR (2) It's not.

I've seen a few (2)s from the inside. I have never seen a (1). Therefore I lean more probabilistically toward the belief that a company is needlessly big.

What surprises me is that that's not where people on this thread are leaning---but has anyone on this thread actually ever seen a (1)??? If not, where is this belief coming from? I've never seen a ghost, so I don't believe in ghosts....

10y agoHN ↗

Consider, for the sake of argument, the possibility that perfect efficiency is unachievable. Almost physically impossible. As a company gets larger, certain kinds of inefficiency in hiring and utilization become absolutely inevitable. With that premise in mind, how many employees does a company "need"? If it would need fifty in that universe of perfect inelastic spheres with no friction or similar effects, how many times that does it actually need in the real world of such efficiency-robbing factors?

In my experience, the answer is two at best and more often, around five. The incremental work product from each new hire diminishes more rapidly than many realize, but the incremental revenue might still make such hires worth it. As Dan points out, a few percent increase in efficiency from optimization can be huge. It might take multiple people quite a while to achieve that, and it will still be worth it. There might be another group of several people trying something else that doesn't pan out, but that doesn't mean they were unproductive dead weight either. Bigger companies have to take risks too. They just take them differently, and the cost is more obviously attributed to them instead of the market.

Maybe a single startup could do what Twitter (for example) does, in fairly short order. Even that's unlikely when issues of scale etc. are considered, but that's not even the point. That single startup is not a valid point of comparison, because it doesn't count the cost of failed experiments. Add up all of the startups trying to do what Twitter does, including the failures, and that probably be a better estimate of how big Twitter (Uber, Netflix, whatever) needs to be.

10y agoHN ↗

Well, I think one of the best points the article made was essentially: A company will continue hiring more people as long as the extra profit from an engineer outweighs the cost.

This isn't literally true in all cases, but it points out that the business decision of how large to grow is based more on that sort of reasoning than "what's the minimum team that can get X done", and maybe a bit different than the analysis you propose.

10y agoHN ↗

At an Alexa rank of 676, that's pretty impressive. I think the key to DDG's success with a small set of engineers is their small scope.

If they ever wanted to expand their product scope, for example, index their own searches instead of aggregation, they would start running into the obstacles discussed in the article. The Lucene comparison in the article doesn't apply to DDG (yet).

10y agoHN ↗

As of 2008, the core search team at Google was just under 100 people.

Then came search customization. Suddenly, no more cached replies.

10y agoHN ↗

100 people?! I could do that myself, in a weekend.

10y agoHN ↗

>There’s also a wide body of research that’s found that decreasing latency has a roughly linear effect on revenue over a pretty wide range of latencies and businesses.

Anyone have pointers to such research?

10y agoHN ↗

This was an error by the author of the blog post. There isn't a wide body of research confirming this. There aren't academic papers. There is merely anecdotal evidence/marketing material from industry participants.

10y agoHN ↗

There is no published research I'm aware of. However, you can find an article about every major tech company talking about how reducing their latency increased whatever their core metric was (not necessarily linearly though, sometimes it was and even steeper curve). I know for sure this is true at eBay, Netflix, and reddit because I've seen the numbers personally, and I have friends at Google, Amazon and Facebook who have said the same.

10y agoHN ↗

Came to the comments looking for these links, too.

Results of my research are below; everything I found is anecdotal, consistent with the other comments. The most robust is the presentation at Velocity in 2009. [4, 5]

My notes:

Small ecommerce vendor, 2012. One second delay in page-load causes 7% loss in customer conversions. [1]

Marissa Meyer, 2006 Web 2.0 conference. Extra 1/2 second in page load time dropped traffic by 20%. [2]

Greg Linden, 2006 (same link as above) "We had similar experience at Amazon.com. We tried delaying page in increments of 100ms and found very small delays result in substantial and costly drops in revenue. [2]

Greg Linden, Make Data Useful, 2006 - Every 100ms delay costs 1% of sales. [3]

Velocity, 2009. The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search Presentation. Multiple observations. My net: Delays under 500m measurably impact customer satisfaction metrics, magnitude increases commensurately with delay. (Microsoft, Amazon, Google) [4, 5]

1 - https://info.ensighten.com/rs/ensighten/images/just-one-seco...

2 - http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20....

3 - http://www.gduchamp.com/media/StanfordDataMining.2006-11-28....

4 - http://radar.oreilly.com/2009/06/bing-and-google-agree-slow-...

5 - http://conferences.oreilly.com/velocity/velocity2009/public/...

10y agoHN ↗

I call this the difference between building a product and a business.

It's relatively easy to build a product and even to ship it.

The real art in building a business to support it.

10y agoHN ↗

The other side of the coin is that big companies do move slow.

There is a lot of communication overheads, a lot of bureaucracy, backwards and cross compatibility concerns, conservatism, wasted duplicate effort, wasted pointless effort, some people who aren't contributing much etc. etc.

This is why tiny startups can sometimes dethrone a large leader. However that's often a multi-man-year effort.

So I'd rephrase the statement: "Why's that company so big? I could do that with 20 brilliant engineers and 18 months". I think that order of magnitude can attack pieces of any established software business from a technical perspective. Then there's obviously the business side.

10y agoHN ↗

That would get you your first launch product maybe. Not an actual enterprise product. Sure, people forgive the small startup for their crashes, their unfinished and unpolished app, their weak customer support, their performance hiccups, etc. As long as they bring something that solves an actual new problem that isn't solved by another product easily available. But an enterprise product, no way you will make that in 18 months with your 20 people while keeping up the maintenance burden you introduced in v1 when you took some shortcuts.

10y agoHN ↗

Hmm somehow it reminded me of brook's law CEO:

"Give me 9 women and I'll give you a baby in 1 month"

10y agoHN ↗

On desktop I reflexively zoomed in to %125 and didn't think about it until I read your comment.

It also has a reasonable default font size my phone, so it's easy to read quickly. This is pretty rare since browsers gave up on text reflow.

10y agoHN ↗

I begin to think that nowadays websites are just designed for "zooming in" in mind (very small fonts, too many space etc.).

So maybe we shouldn't complain. These websites are just meant to be zoomed.

Although some web developers are cruel and block zoom with meta tag (I never understood why some developers did that, maybe for trolling people with poor sight?).

10y agoHN ↗

Pretty simple, we won't bother reading, I just read the comments instead of trying to read that awful formatting.

7 fucking declarations.

Would have been nice

10y agoHN ↗

Businesses that actually care about turning a profit will spend a lot of time (hence, a lot of engineers) working on optimizing systems

Hm - every time I suggest optimization (of anything), I'm shot down with "premature optimization is the root of all evil!"

10y agoHN ↗

What exactly is the point here? At scale, it's not the tech that got the product to the finish line.

The endurance part of the company has nothing to do with tech or even scaling or optimizing that tech. Not even close.

10y agoHN ↗

If your problem is small, viable solutions to that problem are vast. If your problem is large (scale), there's usually only one or close to only one way to solve it. So choosing the right path is important. Since you can't predict product market fit (you cannot predict value), you have to figure out how to iterate and test as close to continuously as possible. This last thing is what most people just can't do. Programmers included. So you can see there's a technology side and a culture side and they have to be well-aligned.

10y agoHN ↗

The question I was wondering about is: When does an organization realize and say, we have hired enough engineers to support our current systems?

10y agoHN ↗

Glad to see some sanity for once and people that realize building 'trivial' apps like Twitter and Uber are not feasible 'in a weekend'. Just making sure the app is secure takes months, and security becomes more complex as features are added. Pretty much every feature you add to the app can explode the complexity exponentially.

e.g: a sorted timeline if strings is easy, but add the simple feature of 'retweeting' and being able to 'unretweet' for example and things quickly become more complex.

10y agoHN ↗

Even if you're just building an open source app to give away, sure, you can get 99% of it done in a weekend. The last 1% will then consume 99% of your time (if you want anyone to use that app).

10y agoHN ↗

I doubt most who say this sort of thing mean it literally (although no doubt some naive sorts do, but they usually are self-identifying through the other outrageous things they say, and thus can be ignored). But one counterpoint to keep in mind is that the alternative view is every company is running as efficiently as possible. I doubt anyone really always believes that, either. For example, when a story about Twitter was posted on HN recently, it was widely discussed that they probably don't need anywhere near as many engineers as they actually have, and could perhaps reduce costs to the point where they are at or near profitability just by reducing staff. There might be some truth to that. Point being, it's not that simple, and there are no absolutes.

10y agoHN ↗

The technology of a big company is often a necessary condition, but it's seldom (if ever) a sufficient one.

We technically-minded people are often pretty terrible at accurately picturing reality as it exists outside of our respective spheres. Consider:

"Why is Armani a billion dollar company? I could make a beautiful suit in a weekend," said the talented seamster/seamstress.

10y agoHN ↗

To this I draw a 2x2 matrix on a whitboard. In each of the 4 quadrants I write:

"Product and Engineering"

"Ops"

"Sales and Marketing"

"Legal"

Enter tasks and goals for each quadrant. Iterate.

This is the minimum effort required to sustain any app. Developers are mostly only aware of the first quadrant.

10y agoHN ↗

I agree and disagree, I've been with companies and seen how scope can expand as you grow, building internal apps, monitoring, reporting, all kinds of stuff. On the flip side i know a guy that managed a team of 5, the 6 of them total managed the snippet editor in Visual Studio. Six people to manage a snippet editor? Really?

10y agoHN ↗

Recently, I was curious why an org that’s notorious for producing unreliable services produces so many unreliable services. When I asked around about why, I found that that upper management were afraid of sending out any sort of positive message about reliability because they were afraid that people would use that as an excuse to slip schedules. Upper management changed their message to include reliability about a year ago, but if you talk to individual contributors, they still believe that the message is that features are the #1 priority and slowing down on features to make things more reliable is bad for your career (and based on who’s getting promoted the individual contributors appear to be right). Maybe in another year, the org will have really gotten the message through to the people who hand out promotions, and in another couple of years, enough software will have been written with reliability in mind that they’ll actually have reliable services. Maybe.

This sounds like it could be Apple?

10y agoHN ↗

Same thing happens in software development at the library level:

"Why not use [Example OSS Library]?"

"Nah, I can do it in 15 minutes."

"You know, they have [X] contributors, lots of great documentation, and 6 months of work into it."

"Yeah, I know [but whatever.]"

10y agoHN ↗

I get the point that you are making, but if something really only takes 15 minutes to do, I wouldn't necessarily use an open source solution just because one exists.

Depending on the situation, the library may have a ton of extra bloat that your application will never use. Also, recklessly deferring to open source libraries can potentially snowball into something like the left-pad fiasco[1].

[1]: http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm

10y agoHN ↗

Yeah, and I get your point too. Eight lines of code? Yeah, you don't need a library. But usually when someone says "15 minutes" they mean 1 - 2 days. So things of that complexity is when it's obviously you should choose a good OSS lib.

10y agoHN ↗

It depends. In many cases you would be right, in many other cases you wouldn't.

It depends on many factors:

- what is the problem? (some problems are really hard, some problems are trivial and just seem hard)

- is developer experienced enough to build own solution?

- are existing solutions good or badly written?(I often refrained from using existing solution when I looked on bloated code in it).

- is solution suitable to project? ("too much bundle size", "doesn't support X", "not too configurable", "too slow", "too hard to use").

etc.

10y agoHN ↗

One of my favourite tech jokes used to be "Google? I could build that out of wget, grep and baling wire!". The fact that "I could build that out of Lucene and money!" is not intended as a joke gives me pause.

10y agoHN ↗

I really enjoy how this guy's blog refers to research papers. It adds to the level of interest.

10y agoHN ↗

Those numbers are whacked. We use ElasticSearch with >10B docs ingested and the cost is like a few thousand dollars a month for hosting.

10y agoHN ↗

From the article:

There’s also a wide body of research that’s found that decreasing latency has a roughly linear effect on revenue over a pretty wide range of latencies and businesses.

I'd be interested in seeing that citation list, particularly for non-google/amazon businesses.

10y agoHN ↗

I'm amused the central topic of this post is that google indexes 50M primes in random files in github, which they seem to not. 961748729 is one of the primes just before the 50M'th

Your search - site:github.com 961748729 - did not match any documents.

10y agoHN ↗

Sometimes it takes a lot of time to reach something that is relatively trivial once you know how to do it.

I can reproduce E=mc² in a matter of minutes. Doesn't mean I could have ever come up with it in the first place.

10y agoHN ↗

we want to maintain an artisanal search index of 1B documents. Then our cost comes down to $12M/yr.

Yeah, no. It doesn't cost that much to maintain an index of that size. Not even remotely close. My roommate coded a search engine back in 2001-02 as part of a class project/hobby and it easily had an index that large, probably larger till he shut it down. The key lies in not crawling it all in a single day. And your index doesn't populate overnight, it takes months of slow crawling. You can maintain an index of practically unlimited size for chump change (well, compared to the millions OP was throwing around). Do people think Excite, Hotbot, Lycos and Yahoo! in the early years had $12 million per year to spend on their indexing? Hell no. Every single one of them would have went bankrupt in a week. Those companies weren't even worth that much back in the mid to late 90s. Your biggest cost is going to be user loads on your servers (CPU/Ram/Bandwidth), not maintaining an index.