Alternately, IAM brings AWS customers a lot of value.
Do you think the average enterprise could develop a unified system for access control to cloud resources that would have a prayer of being secure? Would the average enterprise be able to develop such a system (secure or not) that wouldn't bring work to halt?
The author says as much. The title is clickbait, but the article basically says you're locked into AWS no matter what, and arguing over stuff like Lambda lock-in is pointless considering the other avenues AWS already has you on (like IAM).
Please don't insinuate that someone hasn't read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."
It's a back and forth. You get excited about building lambda functions and are surprised at how easy they are, and how much autonomy you have as a developer. Then, you try to deploy them, and realize you have to file 14 tickets to get the correct IAM settings setup for your production function. No one knows all the permissions you need, and the only way to understand is to try one, see if it works, and what other services you need access to. And, it'll kill all sense of autonomy you had a few hours prior.
I wish people would speak about this problem more. I can’t tell you how many times I need reiterate certain things with our DevOps team to get the proper permissions because I can’t seem to get it right the first time.
If the dev account role has different permissions, it doesn't matter. Of course (s)he tested it first in a dev account. The point is the context around roles matters much more than the code that runs inside it, a critical flaw of AWS. Fairly, this problem is definitely not just limited to AWS but I find their implementation particularly poor.
You should have a custom role for each set of lambdas that make up a service. Your role and your lambdas should be in the same CF template that can be deployed together. Security best practices is for the role to have the least amount of privileges the lambda needs. Not just one generic “lambda role”.
Whoever is responsible for production deployment could audit the role’s permissions and the definitions would be in source control.
I work in a strict environment where the IAM role is not getting deployed with the app. Developers don't have permission to add/modify IAM roles at all. A different team is handling IAM.
I understand that, why wouldn’t the team that handles approving IAM also be approving deploying the app? I’m not saying developers should be willy nilly deploying IAM permissions. I’m saying that having one IAM role that is used by all lambdas is the opposite of a strict environment. The IAM role should be deployed and approved with the resources that need it.
I agree. I still think the whole IAM role thingy could be very tricky. For instance you need a specific role to be able to run your Lambda in a VPC. Let's be honest, it's not extremely obvious when you're new to AWS.
There is always a trade off between security and ease of use. Especially if you want to enforce the concept of least privilege - which you should be doing in a “strict environment”. If you have one role that has any permission that any lambda might need, your entire organization is doing nothing more than security theatre.
If you want ease of use, give all of your roles Admin rights. Is that a good idea? Of course not. But it would also not be a good idea for AWS not to by default deny everything and leave it up to the implementation to decide the level of permissions.
Also, if roles and permissions aren’t the same between the dev and production account, someone must be going in and manually changing permissions either in dev, prod or both. A “strict” environment would have an automated, auditable process to change roles through a pipeline using cloud formation templates that first deployed changes to dev and then after the right approvals through to production.
Our DevOps guy and I spent an afternoon dealing with this. Turns out the AWS Management Console makes it stupidly easy to accidentally overwrite permissions.
Sounds like a process problem. Why don’t you have a CloudFormation template, with the IAM role you need and the lambda function and have it deployed in DEV Account that you hand off to the powers that be?
You also can have the opposite problem where devs have no oversight and there are wide open security groups and IAM roles shared between like 30 lambdas and changing one thing with one of them causes everything to come toppling down like a house of cards because handing that much autonomy to someone who doesn't fully understand (or care about) the underlying infrastructure is like giving a drunk toddler a bucket of hand grenades.
I'm referring, of course, to AWS IAM, which absolutely is the worst form of cloud lock-in that currently exists, at least relative to the zero people who seem to be talking about it.
I assume people don't talk about it because it's not trivially solvable. While we can have many abstractions out there, abstracting IAM across providers is very difficult because while you could get roles and users/members/policies abstracted, you couldn't get resources and permissions abstracted. They are too bespoke. The best you could hope for is a common API, but you'll still be provider-specific in everything you pass.
The only people worried about vendor lock-in are people who have practically nothing of value to be locked-in. "Lambda is horrible for lock-in, once you're hitting billions of invocations a month you're going to wish you could cost optimize by shopping around with other providers! I've got my personal website hosted there, it gets a dozen hits a month, sure glad I'm inside the free tier."
I've heard CTOs/CIOs of large companies express concern over it, but at the end of the day they'll sign that $100M contract with Amazon or Azure. It doesn't actually matter. Its a concern, but its not going to stop the sale or development.
Today, if you're not locked in, you're leaving business value on the table. I hope that changes in the future, and maybe Kube will be the standard platform we've needed to push it forward (for example, I wish I could tell kube "give me a queue with FIFO and exactly once delivery", it knows what cloud provider you're on, if you're on AWS it provisions an SQS queue, if you're on GCloud it errors because they don't have one of those yet, and in either case I communicate with it from the app using a standard Kube API, not the aws-sdk).
But for now, lean in. Don't fight the lock-in; every minute you spend fighting it is a minute that you should be spending fighting your competition.
Vendor lock-in isn't something anyone, especially at C-level should be ignoring. Yes, there are things to be gained and money to be made,but one should always have some sort of alternative,just in case. I happen to be in a business,which 100% relies on one vendor,which keeps hiking up prices every year way above inflation.
The flip side of vendor lock-in — at the megacorp level, anyway — is the “one throat to choke” principle; sure, it might be hard to migrate off AWS/Azure/IBM, but (a) your technology investments have a timespan measured in decade-long iterations; and (b) what you really want is to ensure reliable operations for your enterprise. Sure, you may be stuck with cloud vendor x for the next ten years, but at least you know who to scream at if the infra goes south — preferably someone who can mobilize enough engineers to get you back up and running right now.
Having said that, if you’re a small company where cloud budgets are a significant portion of your overall spend, or a tech company that has a defensive need to be able to migrate providers, YM will definitely V.
Is "migrating from Oracle" the Enterprise equivalent of nuclear fusion? We're always 20 years away from production nuclear fusion, Enterprises are always a few quarters away from migrating out of Oracle %list_your_appliance_here%?
At the end of the day they need to increase revenue by adding customers or by increasing revenue per customer. Once the market is saturated they either buy competitors or they raise prices or try and sell you additional services...
I actually am a fan of the AWS CDK for this reason, and its concept of "constructs".
My big application has a dozen packages, one of which defines generic cloud constructs that host and deploy the other packages as a distributed service.
If we wanted to switch clouds, we'd just have to re-implement the constructs we use for a new cloud; generic things like "ingress data pipeline", which contain dozens of AWS resources each, but have a semantic grouping and defined permissions boundaries.
The visibility timeout begins when Amazon SQS returns a message. During this time, the consumer processes and deletes the message. However, if the consumer fails before deleting the message and your system doesn't call the DeleteMessage action for that message before the visibility timeout expires, the message becomes visible to other consumers and the message is received again. If a message must be received only once, your consumer should delete it within the duration of the visibility timeout.
If a consumer receives a message and then loses its network connection to SQS, the visibility timeout will expire before it can delete the message and it will be delivered a second time to another consumer.
I would very much suggest reading through the linked article by GP and maybe more about the Byzantine Generals Problem which establishes why exactly once delivery isn't just hard, but impossible. And Amazon's marketing department can't change that.
Or you could post the quote from the link I posted.
You can now use Amazon Simple Queue Service (SQS) for applications that require messages to be processed in a strict sequence and exactly once using First-in, First-out (FIFO) queues. FIFO queues are designed to ensure that the order in which messages are sent and received is strictly preserved and that each message is processed exactly once
However, if the consumer fails before deleting the message and your system doesn't call the DeleteMessage action for that message before the visibility timeout expires, the message becomes visible to other consumers and the message is received again.
that you fail to understand.
The phrase "received again" means that the message is delivered more than once. More than once is more than exactly once.
The quote you are referencing is marketing nonsense.
If you want the message to be only received once regardless of whether it was successful processed, you set the MaxReceive count to 1 and set up a dead letter queue as part of the redrive policy and you use fifo queues instead of the standard queue.
It isn’t “marketing nonsense”. It’s more just quoting from one site on the Internet, instead of understanding the system as a whole.
Ah yes, that age old showdown: the fundamentals of distributed systems vs a shiny marketing page.
It doesn’t make what they say any more true. SQS is at least once delivery, and the fact you think otherwise is down to great marketing and maybe a bit of Dunning–Kruger effect.
SQS is only meant for multiple publishers and a single subscriber or set of subscribers that do the same thing. A message in an SQS queue is either delivered or in the queue. SQS operates on a strictly polling model.
Meaning you’re not going to put a message directly in an SQS queue and have it processed by multiple types of subscribers.
Are you expecting some generic system to make sure your subscribers are up and running and polling?
Better question is, do you know what SQS is and have you ever done anything with it?
Exactly once semantics: even if a producer retries sending a message, it leads to the message being delivered exactly once to the end consumer.
This is done with AWS FIFO queues by:
Unlike standard queues, FIFO queues don't introduce duplicate messages. FIFO queues help you avoid sending duplicates to a queue. If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn't introduce any duplicates into the queue.
So exactly once is handle by the producer, the queueing service, and the subscriber.
If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn't introduce any duplicates into the queue.
But what if you try to send and at just about the same time the network fails for 6 minutes? The producer then has the option to try to send again, which may produce a duplicate, or, to assume the first send worked, which may mean the message is lost.
In a system that supported true "exactly once" guarantees, the producer wouldn't have to pick between a duplicate message or a lost message, regardless of the length of network outage.
Apologies, I misspoke. SQS does have a FIFO+Exactly Once Processing mode, so it will never introduce duplicate messages into the queue. It of course doesn't have exactly once delivery without additional state tracking on the consumer outside of the queue.
Alternately, IAM brings AWS customers a lot of value.
Do you think the average enterprise could develop a unified system for access control to cloud resources that would have a prayer of being secure? Would the average enterprise be able to develop such a system (secure or not) that wouldn't bring work to halt?
The author says as much. The title is clickbait, but the article basically says you're locked into AWS no matter what, and arguing over stuff like Lambda lock-in is pointless considering the other avenues AWS already has you on (like IAM).
The average enterprise would buy something like SailPoint which claims to provide IAM for tons of things including AWS.
like MS/Azure Active Directory.
Did you even read the article?
Please don't ask that. It's in the HN guidelines:
– https://news.ycombinator.com/newsguidelines.html
That comparison isn't something the article mentions; it is the article.
Might be nice if the author understood the difference between the CEO and CTO before he went off insulting their positioning.
It's a back and forth. You get excited about building lambda functions and are surprised at how easy they are, and how much autonomy you have as a developer. Then, you try to deploy them, and realize you have to file 14 tickets to get the correct IAM settings setup for your production function. No one knows all the permissions you need, and the only way to understand is to try one, see if it works, and what other services you need access to. And, it'll kill all sense of autonomy you had a few hours prior.
I wish people would speak about this problem more. I can’t tell you how many times I need reiterate certain things with our DevOps team to get the proper permissions because I can’t seem to get it right the first time.
So you didn’t test it in a dev account first?
If the dev account role has different permissions, it doesn't matter. Of course (s)he tested it first in a dev account. The point is the context around roles matters much more than the code that runs inside it, a critical flaw of AWS. Fairly, this problem is definitely not just limited to AWS but I find their implementation particularly poor.
That’s still the issue.
You should have a custom role for each set of lambdas that make up a service. Your role and your lambdas should be in the same CF template that can be deployed together. Security best practices is for the role to have the least amount of privileges the lambda needs. Not just one generic “lambda role”.
Whoever is responsible for production deployment could audit the role’s permissions and the definitions would be in source control.
I work in a strict environment where the IAM role is not getting deployed with the app. Developers don't have permission to add/modify IAM roles at all. A different team is handling IAM.
I understand that, why wouldn’t the team that handles approving IAM also be approving deploying the app? I’m not saying developers should be willy nilly deploying IAM permissions. I’m saying that having one IAM role that is used by all lambdas is the opposite of a strict environment. The IAM role should be deployed and approved with the resources that need it.
I agree. I still think the whole IAM role thingy could be very tricky. For instance you need a specific role to be able to run your Lambda in a VPC. Let's be honest, it's not extremely obvious when you're new to AWS.
There is always a trade off between security and ease of use. Especially if you want to enforce the concept of least privilege - which you should be doing in a “strict environment”. If you have one role that has any permission that any lambda might need, your entire organization is doing nothing more than security theatre.
If you want ease of use, give all of your roles Admin rights. Is that a good idea? Of course not. But it would also not be a good idea for AWS not to by default deny everything and leave it up to the implementation to decide the level of permissions.
Also, if roles and permissions aren’t the same between the dev and production account, someone must be going in and manually changing permissions either in dev, prod or both. A “strict” environment would have an automated, auditable process to change roles through a pipeline using cloud formation templates that first deployed changes to dev and then after the right approvals through to production.
Our DevOps guy and I spent an afternoon dealing with this. Turns out the AWS Management Console makes it stupidly easy to accidentally overwrite permissions.
Sounds like a process problem. Why don’t you have a CloudFormation template, with the IAM role you need and the lambda function and have it deployed in DEV Account that you hand off to the powers that be?
You also can have the opposite problem where devs have no oversight and there are wide open security groups and IAM roles shared between like 30 lambdas and changing one thing with one of them causes everything to come toppling down like a house of cards because handing that much autonomy to someone who doesn't fully understand (or care about) the underlying infrastructure is like giving a drunk toddler a bucket of hand grenades.
I assume people don't talk about it because it's not trivially solvable. While we can have many abstractions out there, abstracting IAM across providers is very difficult because while you could get roles and users/members/policies abstracted, you couldn't get resources and permissions abstracted. They are too bespoke. The best you could hope for is a common API, but you'll still be provider-specific in everything you pass.
The only people worried about vendor lock-in are people who have practically nothing of value to be locked-in. "Lambda is horrible for lock-in, once you're hitting billions of invocations a month you're going to wish you could cost optimize by shopping around with other providers! I've got my personal website hosted there, it gets a dozen hits a month, sure glad I'm inside the free tier."
I've heard CTOs/CIOs of large companies express concern over it, but at the end of the day they'll sign that $100M contract with Amazon or Azure. It doesn't actually matter. Its a concern, but its not going to stop the sale or development.
Today, if you're not locked in, you're leaving business value on the table. I hope that changes in the future, and maybe Kube will be the standard platform we've needed to push it forward (for example, I wish I could tell kube "give me a queue with FIFO and exactly once delivery", it knows what cloud provider you're on, if you're on AWS it provisions an SQS queue, if you're on GCloud it errors because they don't have one of those yet, and in either case I communicate with it from the app using a standard Kube API, not the aws-sdk).
But for now, lean in. Don't fight the lock-in; every minute you spend fighting it is a minute that you should be spending fighting your competition.
Vendor lock-in isn't something anyone, especially at C-level should be ignoring. Yes, there are things to be gained and money to be made,but one should always have some sort of alternative,just in case. I happen to be in a business,which 100% relies on one vendor,which keeps hiking up prices every year way above inflation.
The flip side of vendor lock-in — at the megacorp level, anyway — is the “one throat to choke” principle; sure, it might be hard to migrate off AWS/Azure/IBM, but (a) your technology investments have a timespan measured in decade-long iterations; and (b) what you really want is to ensure reliable operations for your enterprise. Sure, you may be stuck with cloud vendor x for the next ten years, but at least you know who to scream at if the infra goes south — preferably someone who can mobilize enough engineers to get you back up and running right now.
Having said that, if you’re a small company where cloud budgets are a significant portion of your overall spend, or a tech company that has a defensive need to be able to migrate providers, YM will definitely V.
There are loads of orgs trying to get off Oracle as we speak.
Lots of people were already trying ten years ago, and had wanted to try it twenty years ago!
Is "migrating from Oracle" the Enterprise equivalent of nuclear fusion? We're always 20 years away from production nuclear fusion, Enterprises are always a few quarters away from migrating out of Oracle %list_your_appliance_here%?
I'm the progress of doing that right now. It's sweet. So sweet.
At the end of the day they need to increase revenue by adding customers or by increasing revenue per customer. Once the market is saturated they either buy competitors or they raise prices or try and sell you additional services...
Don't forget you can play them against each other. Netflix and Snapchat both have contracts with AWS/GCP
I actually am a fan of the AWS CDK for this reason, and its concept of "constructs".
My big application has a dozen packages, one of which defines generic cloud constructs that host and deploy the other packages as a distributed service.
If we wanted to switch clouds, we'd just have to re-implement the constructs we use for a new cloud; generic things like "ingress data pipeline", which contain dozens of AWS resources each, but have a semantic grouping and defined permissions boundaries.
But we'll burn that bridge when we come to it.
https://bravenewgeek.com/you-cannot-have-exactly-once-delive...
Done
https://aws.amazon.com/about-aws/whats-new/2016/11/amazon-sq...
Well, not quite. What SQS actually offers is: exactly once delivery, except when the message is delivered multiple times.
Quoting from the SQS docs [1]:
If a consumer receives a message and then loses its network connection to SQS, the visibility timeout will expire before it can delete the message and it will be delivered a second time to another consumer.
I would very much suggest reading through the linked article by GP and maybe more about the Byzantine Generals Problem which establishes why exactly once delivery isn't just hard, but impossible. And Amazon's marketing department can't change that.
[1] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
Or you could post the quote from the link I posted.
You can now use Amazon Simple Queue Service (SQS) for applications that require messages to be processed in a strict sequence and exactly once using First-in, First-out (FIFO) queues. FIFO queues are designed to ensure that the order in which messages are sent and received is strictly preserved and that each message is processed exactly once
What exactly is it about:
that you fail to understand.
The phrase "received again" means that the message is delivered more than once. More than once is more than exactly once.
The quote you are referencing is marketing nonsense.
If you want the message to be only received once regardless of whether it was successful processed, you set the MaxReceive count to 1 and set up a dead letter queue as part of the redrive policy and you use fifo queues instead of the standard queue.
It isn’t “marketing nonsense”. It’s more just quoting from one site on the Internet, instead of understanding the system as a whole.
That would make it at most once delivery.
Ah yes, that age old showdown: the fundamentals of distributed systems vs a shiny marketing page.
It doesn’t make what they say any more true. SQS is at least once delivery, and the fact you think otherwise is down to great marketing and maybe a bit of Dunning–Kruger effect.
And I posted above how you can ensure “at most” once - send it to a dead letter queue by setting the retrycount to 1...
Again, understanding how things work takes more than just reading a single web page.
So, uh, what happened to exactly once?
SQS is only meant for multiple publishers and a single subscriber or set of subscribers that do the same thing. A message in an SQS queue is either delivered or in the queue. SQS operates on a strictly polling model.
Meaning you’re not going to put a message directly in an SQS queue and have it processed by multiple types of subscribers.
Are you expecting some generic system to make sure your subscribers are up and running and polling?
Better question is, do you know what SQS is and have you ever done anything with it?
This is one definition of “exactly once”.
https://www.confluent.io/blog/exactly-once-semantics-are-pos...
Exactly once semantics: even if a producer retries sending a message, it leads to the message being delivered exactly once to the end consumer.
This is done with AWS FIFO queues by:
Unlike standard queues, FIFO queues don't introduce duplicate messages. FIFO queues help you avoid sending duplicates to a queue. If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn't introduce any duplicates into the queue.
So exactly once is handle by the producer, the queueing service, and the subscriber.
But what if you try to send and at just about the same time the network fails for 6 minutes? The producer then has the option to try to send again, which may produce a duplicate, or, to assume the first send worked, which may mean the message is lost.
In a system that supported true "exactly once" guarantees, the producer wouldn't have to pick between a duplicate message or a lost message, regardless of the length of network outage.
Apologies, I misspoke. SQS does have a FIFO+Exactly Once Processing mode, so it will never introduce duplicate messages into the queue. It of course doesn't have exactly once delivery without additional state tracking on the consumer outside of the queue.
I'm not worried about AWS IAM. I'd love for my problem to be that small.
I'd like to get off Azure SSO.