There are always nice, but there are a lot of scaffolding work involved with larger systems. It would be nice to be able to provide for example the terraform of the infrastructure and let these tool create their own. Might be a nice feature to add because I haven't seen it done yet.
AWS is still way ahead in cloud usage, that's probably why AWS is the primary target. Anyway, floci has GCP and Azure versions, how mature they are, I'm unsure. But it does exist.
I'm launching something in that space this week - if you're interested in trying one of those out, feel free drop me a message via my website (in bio) :)
I always welcome warmly projects like these, the only problem is that unless there’s a strong community behind, founders tend to build companies around them, raise capital, then put useful functionalities behind paywalls (see localstack).
When I see that, it always reminds me of Steve Jobs’s quote: ”It’s not a product, it’s a feature”.
Aren’t cloud companies like AWS, Azure, GCP the first to be interested in providing their users a meaningful way to do test integrations of their products? Shouldn’t they provide them in the first place?
You’d think that but it doesn’t seem to be the case. I do a lot of offline development and the only way it used to be possible was localstack but that’s AWS only. The open source community is the only interested party, it seems. Thankfully, there’s a lot of contribution happening in this space. Because I have need, I make a lot of my own contributions. I’m doing full stack cloud development against GCP without ever leaving my device.
It depends on the company. Azure provides local emulators for some of their products like CosmosDB and Azure Storage.
Depending on what your size and other things, sometimes it's just easier to have dev instances you test against. Like S3 emulator, you could download one of many and test it but unless your test suite is extensive, the costs against S3 test bucket would probably cost you .03 cents for assurances that everything will work against AWS. If that's breaking the bank at the company, you probably have bigger problems.
Like S3 emulator, you could download one of many and test it but unless your test suite is extensive, the costs against S3 test bucket would probably cost you .03 cents
Not sure if your serious.
You picked the simplest service that isn't expensive at all to run. Have you tried with an RDS cluster for example, how does that work?
Does your integration test create a new cluster to use? That's slow to set up.
Does it use an existing one? Now you might have failures due to noisy neighbors and your cost will not be small.
And that's just RDS, as soon as it's not serverless, costs aren't trivial anymore.
I assume for RDS for integration testing, you could just use publicly available software. So stand up MySQL container if your RDS is MySQL. This is integration testing, not load testing so shared RDS in lowers would be fine as well. Noisy neighbors shouldn't be a big deal.
Again, while local cloud emulator is great for cost sensitive customers, at some point, you should test Dev against almost exact environment you will be running Prod on. If not, you are asking for eventual outage in Prod when some slightly different AWS thing bites you.
There’s also MiniStack, which forked from LocalStack after they started breaking developer workflows: https://ministack.org/
Both fakecloud and its website look sloppily vibe-coded, and its “authors” are anonymous. It’s going to take a while for it to earn trust. I’d treat it with suspicion. (Curl-to-shell pipe to install? Ugh.)
It’s a shell script. You can download and read it before you run it. Piping it directly to the shell is reckless.
I’m not sure how your machine is configured but mine has permission boundaries and security policies that make sure programs are behaving properly. I don’t run everything with my personal user context.
If you're about to install an upstream binary with this and you're not reviewing this then it's purely (irrational) vibes to be concerned with the also unreviewed shell script from the same source.
Yeah but at least with a binary you can verify the checksum of a recent official immutable github release rathrr than just trust a script hosted on the website.
For a mock server? Surely a versioned, standalone executable, library/package or docker image makes more sense. Integration tests generally need to be portable and running on CI, you don't wanna be shell-piping whatever exists in the moment.
curl-to-shell is a terrible installation mechanism because it's not easily reversible and I can't tell if any of the assets are signed, or integrity checked, or not.
The problem, for me, is that self-running installers can create a mess that's hard to keep track of.
I've run Linux without meaningful package management, as that was kind of the style of the time 30 years ago with Slackware. It can quickly become untenable.
There's no real difference between an uninspected script that gets piped straight from the URL into the shell, or a similarly-uninspected make&&sudo make install routine from a tarball. They can both execute code that does bad things (whether unintentionally or deliberately), and they can both leave a mess that is hard to cleaned up.
I've found that it is better to just avoid going down that road to begin with. Whether distro-specific packages, Docker containers, flatpaks, or whatever: All of these make housekeeping easier.
The problem, for me, is that self-running installers can create a mess that's hard to keep track of.
For reasons my machine with a beefy GPU is stuck on an older set of Nvidia drivers. I've got them pinned with apt. The ollama installer fucked everything up by updating stuff that apparently wasn't pinned. After I had fun cleaning up that fucking mess I found it overwrote my custom systemd service file so I had to go in and fix that.
Curl-to-shell is a bullshit antipattern. I have no interest in going back to the dark days of expanding tarballs to / and hoping for the best.
The blog posts are all attributed to "Lucas Vieira" and the dev group https://faisca.dev that is attributed as the author has 2 other projects. Lucas comes up in LinkedIn and looks like an actual person working in San Francisco.
Because taste, creativity, and differentiation matter to a human audience. Perhaps not you personally, but to most people. Ask any experienced marketer.
It’s not really a question of whether they’re using it or not (most of them are); it’s a question of whether they’re using it to generate work product intended to be consumed by their customers. Most know better than that. Some might use it for that, too; but you know what they say about half of the population being below average.
Here are some bugs:
- `ADD m.a.c :inc` update expression wrote a literal top-level m.a.c attribute but didn't change the nested number.
- TransactGetItems returned fields not requested.
- update expression can update the same key multiple times.
In total my conformance suite found 583 failures out of 1279 tests.
Funny fact, DynamoDB local fails about 100 of my conformance tests. Mainly around error message formatting and text. Just be careful about parsing those.
Thank you. I'm going through the DynamoDB issues now, starting with the three you listed. The Parity Suite is great. I'm wiring it into fakecloud's CI.
These miss the point of integration testing (in my Humble opinion). Integration tests are valuable because they get around the problem of mocking. They let you reach out into the foreign thing you use and see if it behaves the way you expect it to. If you replace the real thing with a fake thing, you are, well, by definition, not learning anything about the real thing. It's the subtle behaviors and quirks of real services that often cause the most problems. (It's why testing the DDB Local often hides the painful eventual consistency problems you see with DDB proper.)
My $0.02: just use the AWS SDK to spin up real infrastructure on demand. You can provision SQS in < 10ms. Buckets in <100ms. DDB can take upwards of 30 seconds to a minute if you're adding GSIs, but it's a very small tax in exchange for the confidence it brings. (People often balk at this idea when they first hear it, but give it a shot! I have stamped out many a bug thanks to being able to bounce ideas off real infra.)
Instead of a simulator that runs in its own process, I would recommend mocking out inline in the tests. Only then are your tests entirely “self contained”.
Otherwise your test implicitly make assumptions like “User A is always off-line” (or bucket A is always empty), add a reader of the test would have to go find where that assumption is hardcoded in the simulator and verify it separately from the test itself.
To someone reading that test it is not intuitive that user A is always off-line or bucket A is empty.
If you mock out in line, you can have tests that mock user a / bucket a — to simulate any state for any user or bucket. Perhaps multiple states in one test.
Also think about scenarios like the bucket is initially populated, but then on a subsequent request, it’s empty.
With the simulator approach, you’d have an endless number of permutations of scenarios to add — each of these scenarios would be highly coupled to the test it belongs to, so why not just put it in line in the test itself?
Any reader of the test that uses inline mocking would clearly see which scenario is being tested instead of having to dive into a separate simulator to see what special treatment is hardcoded for User a / bucket a (or maintaining potentially hundreds or thousands of simulated buckets with ad hoc names like “empty-on-second-request” or “bucket–that-responds-slow-and-times-out-but-succeeds-on-retry”.
But how does it compare to floci, which already picked up with localstack left off?
you mean floci? https://floci.io/floci/
Yeah. Stupid really didn’t want to type that. :)
Floci happened to be discussed yesterday, with 47 comments: https://news.ycombinator.com/item?id=49854416
There are always nice, but there are a lot of scaffolding work involved with larger systems. It would be nice to be able to provide for example the terraform of the infrastructure and let these tool create their own. Might be a nice feature to add because I haven't seen it done yet.
I use tf with floci and floci-gcp, weekly…
Oh nice, not aware of this. We're opentofu on GCP, so could be a good addition to ci
I kinda like most of these emulators but why is there none targeting GCP/Azure
AWS is still way ahead in cloud usage, that's probably why AWS is the primary target. Anyway, floci has GCP and Azure versions, how mature they are, I'm unsure. But it does exist.
Right. Floci got posted yesterday here on HN.
https://news.ycombinator.com/item?id=49854416
I'm launching something in that space this week - if you're interested in trying one of those out, feel free drop me a message via my website (in bio) :)
Agreed. Azure would be nice. Something that can work with Terraform too.
Check out floci-az. I haven’t touched it but I know it’s there.
Hard to replicate all the bugs in azure.
I always welcome warmly projects like these, the only problem is that unless there’s a strong community behind, founders tend to build companies around them, raise capital, then put useful functionalities behind paywalls (see localstack).
When I see that, it always reminds me of Steve Jobs’s quote: ”It’s not a product, it’s a feature”.
Aren’t cloud companies like AWS, Azure, GCP the first to be interested in providing their users a meaningful way to do test integrations of their products? Shouldn’t they provide them in the first place?
You’d think that but it doesn’t seem to be the case. I do a lot of offline development and the only way it used to be possible was localstack but that’s AWS only. The open source community is the only interested party, it seems. Thankfully, there’s a lot of contribution happening in this space. Because I have need, I make a lot of my own contributions. I’m doing full stack cloud development against GCP without ever leaving my device.
It depends on the company. Azure provides local emulators for some of their products like CosmosDB and Azure Storage.
Depending on what your size and other things, sometimes it's just easier to have dev instances you test against. Like S3 emulator, you could download one of many and test it but unless your test suite is extensive, the costs against S3 test bucket would probably cost you .03 cents for assurances that everything will work against AWS. If that's breaking the bank at the company, you probably have bigger problems.
Not sure if your serious.
You picked the simplest service that isn't expensive at all to run. Have you tried with an RDS cluster for example, how does that work?
Does your integration test create a new cluster to use? That's slow to set up.
Does it use an existing one? Now you might have failures due to noisy neighbors and your cost will not be small.
And that's just RDS, as soon as it's not serverless, costs aren't trivial anymore.
I assume for RDS for integration testing, you could just use publicly available software. So stand up MySQL container if your RDS is MySQL. This is integration testing, not load testing so shared RDS in lowers would be fine as well. Noisy neighbors shouldn't be a big deal.
Again, while local cloud emulator is great for cost sensitive customers, at some point, you should test Dev against almost exact environment you will be running Prod on. If not, you are asking for eventual outage in Prod when some slightly different AWS thing bites you.
There’s also MiniStack, which forked from LocalStack after they started breaking developer workflows: https://ministack.org/
Both fakecloud and its website look sloppily vibe-coded, and its “authors” are anonymous. It’s going to take a while for it to earn trust. I’d treat it with suspicion. (Curl-to-shell pipe to install? Ugh.)
Agreed with everything up to:
"curl-to-shell pipe to install" - what's the problem here? that's pretty common on linux systems and something the AWS CLI uses.
Or is the problem the fact that this dev is untrusted and is executing a possibly malicious script on your machine?
Running arbitrary code directly in your terminal is very dangerous
Any code I didn't write is arbitrary code. At some point I'm left to trust someone or run no software at all.
It’s a shell script. You can download and read it before you run it. Piping it directly to the shell is reckless.
I’m not sure how your machine is configured but mine has permission boundaries and security policies that make sure programs are behaving properly. I don’t run everything with my personal user context.
So then... Just do that and it's no longer reckless.
If someone wants to be reckless they can be. If someone doesn't, they also have that ability.
Actually... Server can detect if you are piping or not and serve a modified version for inspection.
But really, there is no reason not to use prebuilt packages for distribution. Curlpiping needs to die.
At first I thought “this sounds like bullshit” but then found https://web.archive.org/web/20230408195648/https://www.idont...
Package distribution is the best solution but why would you download the install script a second time?
If you're about to install an upstream binary with this and you're not reviewing this then it's purely (irrational) vibes to be concerned with the also unreviewed shell script from the same source.
Yeah but at least with a binary you can verify the checksum of a recent official immutable github release rathrr than just trust a script hosted on the website.
I'm worried that this behaviour has to be defended.
For a mock server? Surely a versioned, standalone executable, library/package or docker image makes more sense. Integration tests generally need to be portable and running on CI, you don't wanna be shell-piping whatever exists in the moment.
curl-to-shell is a terrible installation mechanism because it's not easily reversible and I can't tell if any of the assets are signed, or integrity checked, or not.
The problem, for me, is that self-running installers can create a mess that's hard to keep track of.
I've run Linux without meaningful package management, as that was kind of the style of the time 30 years ago with Slackware. It can quickly become untenable.
There's no real difference between an uninspected script that gets piped straight from the URL into the shell, or a similarly-uninspected make&&sudo make install routine from a tarball. They can both execute code that does bad things (whether unintentionally or deliberately), and they can both leave a mess that is hard to cleaned up.
I've found that it is better to just avoid going down that road to begin with. Whether distro-specific packages, Docker containers, flatpaks, or whatever: All of these make housekeeping easier.
For reasons my machine with a beefy GPU is stuck on an older set of Nvidia drivers. I've got them pinned with apt. The ollama installer fucked everything up by updating stuff that apparently wasn't pinned. After I had fun cleaning up that fucking mess I found it overwrote my custom systemd service file so I had to go in and fix that.
Curl-to-shell is a bullshit antipattern. I have no interest in going back to the dark days of expanding tarballs to / and hoping for the best.
The blog posts are all attributed to "Lucas Vieira" and the dev group https://faisca.dev that is attributed as the author has 2 other projects. Lucas comes up in LinkedIn and looks like an actual person working in San Francisco.
Re: curl; this seems to work:
faisca.dev is also vibed - perhaps it's a real human using AI to build an online fascade
I don't think it's being questioned that at some point there is human behind this. Though it could always be that spider.
why does it matter that it's a vibe coded website?
Because taste, creativity, and differentiation matter to a human audience. Perhaps not you personally, but to most people. Ask any experienced marketer.
I don't know about your experienced marketers, but mine are all using AI in some way shape or form!
It’s not really a question of whether they’re using it or not (most of them are); it’s a question of whether they’re using it to generate work product intended to be consumed by their customers. Most know better than that. Some might use it for that, too; but you know what they say about half of the population being below average.
As if the MiniStack website wasn't also obviously vibe-coded lol.
It probably is, and that’s not great, either.
Happy living with Moto https://docs.getmoto.org/en/latest/index.html
There are so many of these, but none of them simulate the surprise-billing experience.
File a feature request! ;)
There is also localemu, which I understand to be another localstack fork: https://localemu.cloud/
"46 at true 100% conformance". This is factually incorrect. Inspected their DynamoDB service and it comes nowhere close to implementing the full api.
Here are some bugs: - `ADD m.a.c :inc` update expression wrote a literal top-level m.a.c attribute but didn't change the nested number. - TransactGetItems returned fields not requested. - update expression can update the same key multiple times.
In total my conformance suite found 583 failures out of 1279 tests.
Wow! This is not production-ready at all. DynamoDB would be my main use case. I'll stick with DynamoDB local.
Funny fact, DynamoDB local fails about 100 of my conformance tests. Mainly around error message formatting and text. Just be careful about parsing those.
https://paritysuite.org/ You can check this, awesome work they have to check ddb vs real (local ddb also has a lot of diff with real ddb)
Thank you. I'm going through the DynamoDB issues now, starting with the three you listed. The Parity Suite is great. I'm wiring it into fakecloud's CI.
Local stubs like this are a lifesaver for one-person webhook testing without burning cloud credits every morning.
Interesting thing, looking forward to testing it with my Smithy-generated Python AWS SDK https://github.com/kap-sh/capo
Why not just offer an AWS compatible cloud service at this point ... can you copyright an API?
These miss the point of integration testing (in my Humble opinion). Integration tests are valuable because they get around the problem of mocking. They let you reach out into the foreign thing you use and see if it behaves the way you expect it to. If you replace the real thing with a fake thing, you are, well, by definition, not learning anything about the real thing. It's the subtle behaviors and quirks of real services that often cause the most problems. (It's why testing the DDB Local often hides the painful eventual consistency problems you see with DDB proper.)
My $0.02: just use the AWS SDK to spin up real infrastructure on demand. You can provision SQS in < 10ms. Buckets in <100ms. DDB can take upwards of 30 seconds to a minute if you're adding GSIs, but it's a very small tax in exchange for the confidence it brings. (People often balk at this idea when they first hear it, but give it a shot! I have stamped out many a bug thanks to being able to bounce ideas off real infra.)
You seem to be mixing up integration tests with end to end tests.
Instead of a simulator that runs in its own process, I would recommend mocking out inline in the tests. Only then are your tests entirely “self contained”.
Otherwise your test implicitly make assumptions like “User A is always off-line” (or bucket A is always empty), add a reader of the test would have to go find where that assumption is hardcoded in the simulator and verify it separately from the test itself.
To someone reading that test it is not intuitive that user A is always off-line or bucket A is empty.
If you mock out in line, you can have tests that mock user a / bucket a — to simulate any state for any user or bucket. Perhaps multiple states in one test.
Also think about scenarios like the bucket is initially populated, but then on a subsequent request, it’s empty.
With the simulator approach, you’d have an endless number of permutations of scenarios to add — each of these scenarios would be highly coupled to the test it belongs to, so why not just put it in line in the test itself?
Any reader of the test that uses inline mocking would clearly see which scenario is being tested instead of having to dive into a separate simulator to see what special treatment is hardcoded for User a / bucket a (or maintaining potentially hundreds or thousands of simulated buckets with ad hoc names like “empty-on-second-request” or “bucket–that-responds-slow-and-times-out-but-succeeds-on-retry”.
And if you want to see how they all interact together in AWS, that's why you have a development sandbox AWS account.
The support for SES looks really great, better than LocalStack.
- https://fakecloud.dev/docs/services/ses/