Yeah, debugging 2 bugs that are shadowing each other often takes several orders of magnitude longer to debug than just a single issue.

While having tooling for tight debugging loops allows short release cycles, it leaves that decision up to the business

It works for doctors and lawyers; why shouldn't software developers try it?

It seems like so many "best practices" are really thinly veiled attempts at exploding complexity

I'll bite. :D

It's a bit more nuanced than that IMO, the "deploy often" mantra is only as good as the process around release + deploy. If you half-ass testing and push to production without without a process for verification -- or if, say, your deploy process is half-baked, or your staging environment is worthless -- you can probably expect "interesting" production deploys on a pretty regular basis.

As much as we'd like to pretend we're all good engineers, this happens more often than you'd expect -- even with good engineers: at some point a company transitions from scrappy startup to a shambling beast, and the things that used to work for a scrappy startup (like skimping on testing and dealing with failures in production) are insufficient when you've got more eyes on the product. Further, the engineering culture remains stuck in "scrappy startup" mode long after the shambling has begun.

And all that's ignoring the fact that less frequent deploys with more changes have their own set of problems. We actually got to a point with deploys of a certain distributed system such that we were terrified if we had more than a few days worth of changes queued up. So many things that could go wrong! :)

We create ourselves so many of the problems we are paid to solve.

This, on the other hand, I completely 100% agree with: if not us, then who? :)

EDIT: minor formatting change

Sounds like a testing/UAT issue.

Absolutely. Confounding issues are the biggest problem, even more so than sheer volume in a small time frame.

You can roll back a small release that broke the world. A

Or the opposite: embrace failure, practice resiliency, work with it instead of against it!

Funny, I've taken it as justification for releasing often. If I have a hard time changing one thing without breaking the service, it's nearly impossible to change a hundred things without breaking something. Since it's a given that I will have to change things, I'll try to stick to a scope where I stand a chance of doing so successfully.

No, in the video he states that the weekend is their busiest period.

Absolute numbers of alerts are probably a lot less useful than alerts/use, or alerts/users. After all, if people use the services PagerDuty covers less often on the weekend then you might see a lower alert volume even if issues are relatively more common.

most production issues are caused by people

It's a well known fact, both for systems and networks.

The user experience is pleasant as well. Nothing better than your dependency breaking at 4PM on Friday with no support until Monday.

In the talk, he specifically mentions that weekends are the busiest.

Your statement is true in San Francisco, but Pacific time zone also encompasses Oakland, south bay, los Angeles, and san Diego, where relatively few people commute using Uber compared to going out using Uber.

From personal experience, having driven for Lyft and Uber I'm two cities (Sf, sd) surge in SF does get acutely high in the morning, more broadly in the evening (work departure times are spread out over a wide range of hours), and acute around 2am. In California, bar closures statewide are 2am at the latest, so that accounts for unified departure times in multiple markets.

Would PagerDuty consider publishing some anonymized aggregate statistics?

90% of outages are caused by configuration changes which is why change management was so hot for enterprise software.

Oh I know. I've been in a meeting with several bosses where I tell them flat out "don't give me more developers, give me more hardware and tools". They usually look like I have an extra arm growing out of the top of my head.

That does save a lot of money :-) The trick is managing the rate of change and the risk of disruption. If you manage it to no risk you end up changing too slowly, if you manage it to close to the risk you end up with unexpected downtime and other customer impacting events. Understanding where you are between no risk and certain doom really only comes with experience.

I'm bad at optimism, but maybe the takeaway here is that we're finally at the point where it's not the hardware or systems that are the biggest problem. The automation is actually working well enough that it's not the tall tent pole.

Bet I can!;o)

Question is : people at each level. Here most of the alerts happens at night... due to batchs...

Most of the other alerts are due to network issues, moving servers due to hardware intervention that change the topology of the network, etc etc.

Couldn't this also just as easily be a change in usage patterns?

It's also ok to "move fast and break things" when the most critical thing you're breaking is your annoying high school "friend"s ability to share the latest racist meme with you. There's a reason facebook made this phrase popular.

And it's OSS: https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

Oh perhaps they did mean shaved-ape simulator. I read none of it as derogatory, and hopefully my response didn't suggest otherwise.

I hear what you're saying, in theory. In practice, a system is always changing – regardless of whether developers are deploying code changes.

Some events (or "changes") I've seen in last year that caused a system to stop running fine:

- analyst ran query that locked reporting DB, which had ripple effects

- integration partner "tweaked" schema of data uploaded to us

- NIC started dropping packets

- growing data made backup software exceed expected window, locking files

- critical security patch released (didn't cause a problem exactly, but did change the system from having no known security holes to having a big one)

On and on. So, yeah, I'm not disputing the idea that changes are often the source of problems. I'm just saying that any moderately complex system is constantly encountering new events whether or not the developers are making changes.

This isn't a rebuttal of your statement so much, as it is rebuttal of a common attitude I see in business folks. A lot of non-technical executives seem to have the mentality that software is "done" after it's built, which is naive IMO.

Active systems require active maintenance. You can avoid a lot of problems with intelligent architecture and robust instrumentation/monitoring, but at the end of the day systems rot and will eventually stop running fine.

And even if your perfectly planned and architected system runs in total isolation on a private server where security isn't a major concern, you'll still build up technical debt if you aren't routinely upgrading major libraries, etc. You'll get to a point where you want to implement some feature, and while there are 3 excellent OSS libraries for your platform to accomplish it, none of them is compatible with your 5 year old version.

You probably know all of this and might even agree with most of it. I just had a visceral reaction to the idea that "a system running just fine will usually continue to run just fine," and felt compelled to respond. :)

Note: most of my experience is with public or enterprise web applications. I imagine other types of systems have other problems.

I believe I read it here, though there are several similar studies and this might not be the one I originally read.

https://www.ncbi.nlm.nih.gov/pubmed/25531231

That study gave no causal reasons. In particular, there was no effect on number of surgeries done. It's just as possible that research-oriented doctors go to the conference, and those doctors are worse at performing surgeries.

Isn't that because a surgery, if it fails, will result in the patient dying immediately, rather than some random time later? If you do more surgeries, you will increase both the failed and the successful surgeries, whether your success rate is 10 or 90 percent.

Compare: "bomb explosions tend to increase around the time when we send in the bomb squad. I guess bomb defusing is pointless."

IOW: Surgeries and bomb defusing tend to move forward a lot of the death-probability-mass (while, in theory, destroying some of it).

In an Agile context at least, it means test out ideas and reiterate quickly. Breaking does not necessarily mean breaking existing things in a visible way.

It's a bad analogy because they don't upgrade ships while they are in use. People aren't saying software is most stable when it's not used, they are saying it's most stable when people stop changing it.

Ok, that's a reasonable position, though not one I completely agree with.

It would be nice if we as customers had the freedom to pick up which upgrades to install, and to limit ourselves to security patches and (maybe, some) library deprecations. Windows upgrades let you do just that, even if it is a pain to manage. In my previous job, we held monthly meetings to discuss which windows patches to install in our dozens of business critical Win2k3 servers.

For most software, upgrades are opaque. In the best case, you are given a binary choice: install now or install later. In many cases you dont even get that, either the product stops working until you apply the patch or the patch gets automatically applied without your explicit consent.

Which would itself not be a problem if most upgrades did what they are ment to do. Instead, as the GP suggest, patches introduce new (sometimes critical) bugs all the time. With pure repair patches, the balance is mostly positive siince they fix more bug than they break, but with every new feature (which usually is required by a minority of very vocal customers) comes a risk that some other more important feature is going to stop working as expected.

Industry wide, we lack the discipline to change things in a responsible way. Change management is today what source version control was back in the 90s: something that most people has heard good things about, but most people is not doing correctly, and a significan minority not at all.

Does anyone who has worked in software for any period of time not realize this? Computers are dumb they (generally) repeat the same tasks reliably _compared to humans_. Occasionally there are system failures but most failure are caused by code or configuration changes, especially when they interact with inputs and systems in untested ways.

Depends on how smart your supervisor gets doesn't it but sure, why not? Many times processes can be restarted in microseconds...

I've had uber crash on drivers while on a ride, even 30 seconds is better than complete failure! I think it would make uber more resilient...

Care to explain which bits Erlang/Elixir doesn't do or at least provide a solution for? As the sibling show you may need to be smart about certain things but the OTP definitely provides ways of solving the above things.

That doesn't contradict my conjecture that the issue was improper sharding. The jump in duration of cancellation detection from 2ms to 20ms could have been because they were running that calculation on a single machine.

Although... Ethernet latency would probably make it tough to stick to 2ms.

Sounds like they have 2000 very individualistic engineers with that level of insanity.

My guess (and it is just a guess) is that each of these services is made up of multiple even smaller libraries which are each being version controlled as a distinct entity. I'm not sure I could fully get behind working at that level of granularity, but it would explain how you end up with quite so many git repositories.

From what I understand, they split their microservices into logical packages and libraries, each of which have their own repo. Then some microservices have a separate repo to track configuration changes.

4 repositories per engineer

Is that strange? I've got 6 repos at work for various utilities and "personal" projects I'm working on in addition to the 4 repos for team wide projects.

I would bet that they have location data in git with each major market being it's own repo.

THE HN PC police are back, up to their old antics.

True. I was using it as an argument to adjust the system architecture, as well. My argument was it didn't matter really which changed to get things into alignment, but that having them be different was a bit of a concern.

  10 git repositories per microservice

I'm reminded of http://danluu.com/monorepo/

A leading example of overfragmentation.

On-call shifts are going to be interesting with an average of 2.5 engineers per service, not to mention handling people switching teams or leaving the company.

Sounds like they should sit down and re-think why they need one service for every two engineers (!)

They did that; the reason is job security.

Maybe they prefer repos to branches?

This leads to an interesting perspective, that there is a hard limit on how much traffic your services will ever have to handle, and that limit is probably a constant expression on the planet's population.

But I can't imagine how you could possibly need 8000 git repositories unless you're massively over engineering your problem. Project structure tends to reflect the organizations that build them.

This seems almost the exact opposite of what I've heard of Facebook (i.e.: they have a massive repository).

2000 engineers sounds like overkill for a mobile app

Not really what the video addresses for the most part. Mostly talking about their development process and impact of breaking up their application into 1000s of microservices. This is more to handle their development team scaling, rate of change etc...

you'd still want a globally unique ID for each user (so you could run cross-city analytics).

Analytics can run in batch mode. Also it is read-only operation. So it is far simpler than designing a distributed database.

As an aside, why would Uber want to reinvent a distributed database for storing user information?

To me, it seems the best thing to do in that scenario is taking the huge "shard by city" gain when it comes to rides. Then dealing with the global nature of users by having global user DBs sharded by user ID.

Everything related to driver selection and real time tracking is very specific to one location. The server responsible for tracking drivers in San Jose doesn't need to reference rider locations in NYC under any circumstance.

The things you highlighted are very light load comparatively. Sign in, load account info, then dispatch to location specific shard.

Scaling the traffic is not the issue. Scaling the team and the product feature release rate is the primary driver.

The Netflix stack has a lot of assumptions about Java, and most of Uber is written in languages other than Java.

I can't be the only one who keeps reading it as: Whitest Kids U Know.

Yes, and much more than 2 per instance of a service.

The # of services vs # of engineers isn't really bad though; you just need the tools to make the commonalities between services easy.

Why?

People talk about interpreted languages being slow sometimes, now your program is divided over 1700 separate servers and instead of keeping variables in memory you have to serialize and send them over the network all the time.

Good god, what on earth for?

This just sounds crazy to me. How often do the services that are responsible for a given region change?

Strictly this is not necessarily relevant. You can easily roll that table hit into another db hit you were already making. Can't do that with services.

You're both missing the point here. Both things are conceptually equivalent:

- Select(db, somekey, someparameters) [return some db object]

- http_get_query(http://service.com/somekey/someparameters [return some JSON]

They are external (micro)service:

- they both need the target system to be available.

- they both may fail in weird and unexpected ways.

- they both need to handle failure gracefully.

Their usage have different properties:

- A database call need to have a permanent connection pool to the database, usually requiring db user and password.

- A http call is just call and forget. It's a lot easier to use, in any applications, at any time.

You might want to add functionality such as caching or business rules.

A better question would be why not write a module or class? There a pros and cons to either, but advantages include: better monitoring and alerting, easier deployments and rollbacks, callers can timeout and use fallback logic, you can add load balancing and scale each service separately, it's easy to figure out which service uses what resources, it makes it easy to write some parts of application in other programming languages, different teams can work on different parts of the application independently as long as they agree on an API.

I understand what microservices are, but I can't understand what 1700 pieces of unique functionality of Uber could be abstracted into their own services. I am struggling to even think of 100, so I was curious what exactly some of these things were, and how they structured things to need so many service dependencies.

It looks that for functions as simple as that the RPC overhead should be pretty small, or it will eclipse the time / resources spent by the actual business logic.

E.g. I can't think of a REST service doing this; something like a direct socket connection (over a HA / load-balancing level) with a zero-copy serialization format like Cap'nProto might work.

This type of comment presumes that there is a right number of services for a company to run.

What number is that?

Whoa. That's is an insanely large amount of applications. I'm assuming that's essentially one function per microservice which is one of the huge do-not-do of microservices as it's just a complete waste of time, resources, etc.

I would love to hear a breakdown. This sounds like a nightmare to maintain and test.

And it took 1-3 nights to settle.

And another to redirect the misspelled first attempt "left-padd-as-service" API calls to the new one

I have heard of, but never witnessed, groups where the team stays together but cycles through projects a sprint at a time. One or two teams keeping three or four projects spinning but making the projects take turns.

I don't know if they provide popcorn at the meetings where the project managers explain why they deserve the next sprint.

Nonsense. libc? A microservice for every function call! Surely if you can wrap your head around a library you can wrap it around a thousand little microservices.

(Snark exists in this comment.)

I used to work in startups, and wondered this all the time. Then I joined a big valley tech company, and now I understand.

It's because they hire smart and ambitious people, but give them a tiny vertical to work on. It's a person level problem, not a technical one in my opinion.

I think you solve this by designing your stack and then hiring meticulously, instead of HIRE ALL THE ENGINEERS and then figure it out (which is quite obviously uber's problem)

You went through the slides but didn't watch the video? If so, you can't really make the argument that it isn't interesting.

I'm recommending the video explicitly. Not the slides!

REST/JSON makes sense between a human backed browser and a middle tier; between machines, SOAP is the best contract first technology, or at least a custom XSD backed XML schema.

Assuming you're not omniscient you'll be refactoring regardless. The difference is whether you'll be paying as you go (clients want a JSON API, we need to add new columns to a table but it'll lock rows) or if you'll be taking on technical debt to be repaid in the future (turns out Mongo sucks and we would do much better with Cassandra for serious horizontal scaling).

I believe that if you aren't extremely certain about what the future holds it may be best to work with a more flexible technology first and transition to a more structured setup once you have solved for your problems and identified intended future features. And if you are extremely certain about what the future holds you're either insanely good at your job or just insane.

I think the key is to do regular refactoring as you go -- it has to be an ingrained part of the process. It's really a management issue. Not every company/project has the foresight to budget for this, of course. If a team can't or won't regularly improve their infrastructure, then yes, a more structured approach would probably be better for anything that needs to last.

Of course there are other considerations. A more "planned" structure always makes sense if you're talking about systems or components that are life-critical or that deal with large flows of money. The "fast and loose" approach makes the most sense when you can tolerate occasional failures, but you have to have fast iterations to be quick-to-market with new features.

In my experience the likelihood of your scenario (increasing resistance to refactoring) is almost always indirectly proportional to the amount and quality of refactoring tools available.

500KLOC JVM/.NET application? No big deal.

50KLOC JS/HTML-based SPA? Pfooooh. That could take a while...do we really need to?

This is the key point I think. Flexibility is very important when a project is immature. And as it matures it benefits from additional safety. Too bad so few tools support that transition.

I'll take gross generalizations for $500, Alex.

Protobuf v3 allows use of JSON and gRPC for Node.js handles JavaScript objects quite handily. Might not be mutually exclusive.

It fits really nicely with a data > functions mindset.

That will probably confuse my procedural mind, much like declarative "stuff" often does! :-)

Camp 3: those of us who think "shove something, anything that 'works' out as fast as possible" is a recipe for disaster, irrespective of the transport format or service infrastructure.

I suppose folks in this camp would overlap significantly with those in camp 2, though.

That's a really silly (and in my experience) inaccurate comment.

I like schemas ... most of the time! I like Avro ... some of the time! And JSON some of the time. And I write mostly in Python.... and Scala.

The world is not so black and white.

The third camp is the guys who know that you have to use schemas when it makes sense. :)

I don't understand the conflation of Rest/JSON with NoSQL. The most popular frameworks for the dynamic languages you cite all use SQL with strong schemas out of the box. Come to think of it I'm not sure why we're even associating Rest with JSON specifically or any content type for that matter when one of the major points of Rest is content-negotiation via media types. You can even implement a messaging concept with Protbufs in a strictly "rest-like" fashion.

But you are oftenwrong.

SQL databases should be conservative in what they accept. Once you've inserted crap into the DB, it's hard to fix it.

You can make your schema very light and accepting almost like NoSQL which is how you get into the situation you described; the solution is to use stricter schema. That and it helps to hire a full time data engineer/administrator.

Using NoSQL as a fast cache

I'd rather use caching technology, specifically designed for caching, like Redis or Varnish or Squid.

"Once you've inserted crap into the DB, it's hard to fix it."

Agreed, it is hard to fix. NoSQL databases can be really hard to fix when they are full of crap, too.

You're missing the point where you have a flexible, schemaless design while you're specing out the service and seeing if it's even necessary, viable, and will be needed for a long time for a long-lived system.

Properly encoding rigidity through type systems, SQL checks/triggers/conditions, etc is hard. It takes a really long time to really iron out all the string and integer/double/long typing out of your system, let alone do it in a way which matches up properly with your backing datastore. Once you've got it set up and nailed down with tests, then you're golden, but that's a cost that is usually not worth paying until long-term need is determined.

JSON is not inherently schemaless, it's just generally not used with one.

Initiatives like JSONSchema go some way to restoring some constraints to the format and prevents unchecked deviation over time.

The longer a database has existed, the more likely somebody in the company wrote something crucial that accesses it without your knowledge and without going through your ORM (usually because your ORM isn't implemented for the language they're using, or it emits bad queries for their use case). Sanity checks that aren't enforced by the database can't be relied on to be up to date or even happen at all.

Snowflake does have a relational database engine, it was actually founded by two Oracle engineers (who had worked at Oracle 15+ years)

Spot on

I've worked on both types of projects. My previous project used Scala on the backend and Node for the public API. Thrift was used to maintain persistent, reusable, connections between the public API and the backend, and to enforce strict typing on message values.

What I really liked about Thrift is that all I needed to know how to use the service was the thrift definition file. It was self-documenting.

My current project uses HTTP and JSON to communicate from the public API to the backend services. There is significantly more overhead (latency and bandwidth) and no enforced document structure (moving toward Swagger to help with that).

HTTP+JSON is great for the front-end where you need a more universally parsable response, but when you control the communication between two systems, something like Thrift/Protobuf solves a lot of problems that a common with REST-ish services.

No No No. This is like asking which is a better tool - a nail or screw? Since everyone is so familiar with SQL, they are avoiding very good reasons to use NoSQL.

'NoSQL' is just a broad term for datastores that do not normally use standard Structured Query Language to retrieve data. Most NoSQL do allow for for very structured data as well as some query languages that are similar to SQL.

BigTable (Hadoop, Cassandra, Dynamo) and block stores (Amazon S3, Redis, MemcacheD) are absolutely critical to cloud services. Json tuple document DBs are needed for mobile and messaging apps. Graph is for relationships, and Marklogic has an awesome NoSQL Db focused on XML.

Full disclosure: I am the founder of NoSQL.Org - but I also use multiple relational SQL databases every day.

(3) people who ponder what's the best solution for a given situation? Sometimes you want dynamic languages, or schema-less data. Sometimes you want rigid interfaces and verifiability.

Eg rolling out a public API in Thrift/protobuf will severely difficult its adoption, whereas Rest/JSON is pretty much the standard - but then building microservices that communicate to each other in Rest/JSON quickly leads to a costly, hard to maintain, inconsistent mess.

We're obsessed with "one fits all" absolutes in tech. We should have more "it depends" imho

I got told by an Uber engineer that's because of their hyper-hyper growth their tech is basically the biggest mess possible.

That's also why that makes it an interesting place to work at and helped them achieve this growth.

Personnaly I think this should be made into a global app with no geo-fencing (e.g. Available everywhere basically).

Absolutely amazing to watch. I think most of the big companies (Amazon, Google) already have solutions for these issues like: limited number of languages, SLAs between services, detailed performance metrics and the ability to trace microservices.

Almost as much as the overhead of splitting everything to a thousand microservices...

Really enjoyed this talk. Our services don't quite (yet :)) run at that scale, but many of the issues mentioned have already peaked at some point. It's also good to have (more) validation to some choices we have made in the past, are currently making or are thinking about making in short term future.

Ah... it's almost the reason for GraphQL.

Basically if you're building a hypermedia REST API you return an entity or collection of entities whose identifiers allow you to fetch them from the service like so:

    {"result": "ok!"
     "users": ["/users/123", "/users/234"]
    }

The client, if interested, can use those URLs to fetch the entities from the collection that it is interested in. This poses a problem for mobile clients where you want to minimize network traffic... so you over-fill your data collection by returning the full entity in the collection.

    {"result": "ok!"
     "users": [{"id": 123, "name": "Foo"}, {"id": 234, "name": "Bar"}]
    }

The trade off is that you have to fetch the data for every entity in the collection, the entities they own, etc; and ship one really large response. The client would receive this giant string of bytes even if the client was only interested in a subset of the properties in the collection.

GraphQL does away with this problem on the client side rather elegantly by allowing the client to query for the specific properties they are interested in. You don't end up shipping more data than is necessary to fulfill a query. Nice!

... but the tradeoff there is that you lose the domain representation in your URLs since there are none.

Yeah, that absolutely didn't compute for me either. If it really was the case with no thoughts needed for transactions or correctness I would have started to split my monolithic apps years ago. Its when your data model is complex enough to make consistency/correctness difficult without transactions you have a real trouble splitting things up.

You seem to be repeating these weird myths that have no basis in reality.

No basis at all? I knew I was unhinged...

You don't need to be an expert in distributed systems to use microservices.

True. Hooray for abstractions. You don't need to understand how the V8 engine allocates and garbage collects memory either... well until you do.

It's literally replacing a function call with an RPC call.

You're not wrong.

Which is the point. Whether for architectural or performance reasons I think you need to understand your domain and model your data first. For domains that map really well to the microservice architecture you're not going to have many problems.

And a formal specification is overkill for many, many scenarios. That doesn't mean they're useless. They're just not useful, perhaps, for e-commerce sites.

But anywhere you have an RPC call that depends on external state, ordering, consensus... the point is that the tradeoffs are not always immediately apparent unless you know your data really well.

And I've worked on 2 of the world's top 5 ecommerce sites which both use microservices.

And I've worked on public and private clouds! Cool.

The point was and still is the same whether performance or architecture... think about your data! The rest falls out from that.

RPC vs PC means introduces a greatly increased probability of random failure. If you're trying to do something transactional, you're in a different world of pain.

[Scalability i]s an aspect. It's often the beginning of a micro-service migration story in talks I've heard.

It absolutely helps with people and org scalability. I haven't seen it help with technical load scalability (assuming you were already doing monolithic development relatively "right"; we ran a >$1BB company for the overwhelming majority on a single SQL server).

With micro services you could have different versions of those utilities in use which would not be possible in a monolithic app.

You have no idea...:)

not for everyone

If you run multiple instances of your server and follow certain reasonable practices, you can take down old-version instances one by one and replace them with new-version instances, until you have upgraded the entire fleet.

Alternatively, you can deploy the new version to many machines, test it, then make your load balancer direct the new traffic to the new instances, until all old instances are idle, and then take down these.

And in microservices you can just pick whatever stack you feel is the best for just that task

You certainly can do that. A little discussed benefit (or curse) of a microservice is maintaining an API (as opposed to ABI). Changes are slower and people put more thought into the interface as a discrete concern. I am curious if an ideal team would work better with a lib--I think so, but I'm not sure!

I wonder if there's any writing out there about the evolutionary hierarchy of refactoring, like:

code > function/method > module > class > ... > microservice

Well, granted. It will take decades for we to have decades of experience with microservices that break clear encapsulation.

I think it's worth noting, it doesn't just split the code. It also splits the teams. 5 engineers will coordinate automagically, they all know what everyone is working on.

At perhaps 15, breaking up into groups of 5 (or whatever) lets small groups build their features without stomping on anyone else's work. It cuts, not only coupling in the code, but coupling in what engineers are talking about.

The two sort of obvious risks, those teams probably should be reshuffled from time to time, so code stays consistent across the organization. If that's done with care, specific projects can get just the right people. If it's done poorly, you sort of wander aimlessly from meaningless project to meaningless project.

The other one is overall performance and reliability. When something fans out to 40 different microservices, tracking down the slow component is a real pain. Figuring out who is best to fix that can be even worse.

Oh they definitely do. It's just a different flavor and the debt scales differently. Ideally the debt amortizes across the services so you can solve a problem across multiple places. This is obviously really difficult to discuss or analyze without a specific scenario.

It's probably not worth discussing unless you're actively feeling either perf or coupling debt pressure.

Great video. I went in expecting it to cover mostly the technical side of things. Instead Matt gave a great overview of the team / engineering organization dynamics to watch out for when adopting microservices. (I particularly liked that he pointed out how developers may write new code / services instead of dealing with team politics.)

>> Do you think that's it's that people would rather deal with operational complexity, or just that it's easier to not think about operational complexity early on

That's one way to scale development projects.

Have multiple teams of dev who work on separate stuff. They can develop in their corner however they want and that makes them happy. The final thing is a clusterfuck of services with little cohesion [YET they ALWAYS manage to put the shit together in production in a mostly working state].

The alternative implies to have a consistent and cohesive view of the components. For that, you'll need to have people with experience in architecturing scalable and sane production systems [so they understand what are the consequences and tradeoffs of ALL their decisions]. I know of very few people on the planet who can design systems. (We're talking past unicorn-rarity here). Plus, the developers must actually listen AND care about the long term maintenance AND the people involved (i.e. not want to do shit that will hit either them in 6 months or the team in the next office).

The amount of collaboration AND communication AND skills required to operate this strategy is beyond the realm of most people and organisation. There are very few individuals who can execute at this level.

The inherent greater complexity of designing extensible and cohesive systems turns away a large subset right away and it never gets to a comparative matrix to inform the decision making process. It's been reduced to "monoliths bad, microservices good" level of thinking.

I honestly think that the rise of microservices as "the technical cure for cancer and everything else" is a very interesting surfacing of the various systemic disfunctions of our beloved software industry. The disfunctions have been present from day 1 but the environment has somewhat radically changed.

Exactly. Testing an API in isolation doesn't help you a thing when the bug arises because of, say, subtle inconsistencies between API implementations of interacting services on a long chain.

This sounds a lot like the code coverage fallacy. (to which I usually answer "call me when you have 100% data coverage").

I quite enjoyed watching this. My takeaway isn't so much that this is a critique or endorsement of microservices, but rather just a series of observations. Not even lessons learned in many cases, just honest pragmatic observations. I like how he doesn't seem to judge very much – he obviously has his own opinion of several of these topics, but seem to let that more or less slide in order to not get in the way of these observations.

Good talk, will likely watch again.

Honestly I don't think the problem is microservices. I mean everything he brings up is true, but it's more of a "how you do microservices" issue.

I used to work in startups, and overall was impressed with velocity. Then I joined a big valley tech company, and now I understand.

It's because they hire smart and ambitious people, but give them a tiny vertical to work on. On a personal level, you WANT to build something, so you force it.

I think you solve this by designing your stack and then hiring meticulously with rules (like templates for each micro service), instead of HIRE ALL THE ENGINEERS and then figure it out (which is quite obviously uber's problem)

Yes, I think that is the primary problem - they invested in Zipkin. At the end of his talk, he mentions the idea that you should prioritize core functions and use vendors for everything else. But he's right that some person created that code - and thus it becomes difficult to get them to switch. And instead it becomes a huge source of technical debt for a company that is non-core to their mission.

True - and marketing to developers is very challenging - a lot of noise to get above.

Hey, Matt Ranney, I used your node-pcap library to learn how to parse PCAP :) Did not know you worked for Uber, thanks!

I couldn't agree more.

Yeah, that's completely ridiculous.

I'm currently working with a very (very) large UK-based retailer, with various outlets globally, and I can tell you from first hand experience that their selection process is ruthless. There are no, or at least very few, lamers in the tech team.

And yet, when you look at the systems, and the way things are built, some of it just seems crazy. Only it's not, and for many reasons, of which here are only a couple:

- The business has grown and evolved over time - with significant restructuring occurring either gradually, or more quickly and in a more intentionally managed way; systems have grown and evolved with it, often organically (which is something I see at every company I've worked with).

- Legacy systems that make it really hard to change things, and migrate to newer and better ways of doing things because they're still in production and depended upon by a host of other systems and services.

- The degree of interconnectedness between systems and services is high across the board; this isn't a reflection of bad design, so much as a reflection of the complexity of the business.

In any large organisation these kinds of things are apparent. I would imagine that in a large organisation that has become large very quickly, like Uber, if anything the problems would be magnified.

To say that they have poor engineers is therefore very unfair: these are people who would have needed to ship RIGHT NOW on a regular basis in order to facilitate the growth of the business. That's going to lead to technical compromises along the way, and there's little to be done to avoid it. There's also a difference between being aware of a problem and having time, or it being a priority, to fix it (e.g., iterating and retrieving).

I agree that it is utterly stupid. However, it is almost certainly a political / interpersonal / interteam issue. Very probably, the person making the calls couldn't or didn't convince the service they were calling that a bulk access mechanism was required.

It probably doesn't help that canonical REST as usually applied to APIs doesn't really have a single well-known pattern for this.

Or your making assumptions based on limited knowledge.

When did we loose our heads and think such an architecture is sane?

It's not sane. That's the entire purpose of the talk.