This comes up now and again, somewhat akin to the Rust soundness hole issue. To be fair, it is a legitimate issue, and you could definitely cause it by accident, which is more than I can say about the Rust soundness hole(s?), which as far as I know are basically incomprehensible and about as likely to come across naturally as guessing someone's private key.

That said in many years of using Go in production I don't think I've ever come across a situation where the exact requirements to cause this bug have occurred.

Uber has talked a lot about bugs in Go code. This article is useful to understand some of the practical problems facing Go developers actually wind up being, particularly the table at the bottom summarizing how common each issue is.

https://www.uber.com/en-US/blog/data-race-patterns-in-go/

They don't have a specific category that would cover this issue, because most of the time concurrent map or slice accesses are on the same slice and this needs you to exhibit a torn read.

So why doesn't it come up more in practice? I dunno. Honestly beats me. I guess people are paranoid enough to avoid this particular pitfall most of the time, kind of like the Technology Connections theory on Americans and extension cords/powerstrips[1]. Re-assigning variables that are known to be used concurrently is obvious enough to be a problem and the language has atomics, channels, mutex locks so I think most people just don't wind up doing that in a concurrent context (or at least certainly not on purpose.) The race detector will definitely find it.

For some performance hit, though, the torn reads problem could just be fixed. I think they should probably do it, but I'm not losing sweat over all of the Go code in production. It hasn't really been a big issue.

[1]: https://www.youtube.com/watch?v=K_q-xnYRugQ

It took months to finally solve a data race in Go. No race detector would see anything. Nobody understood what was happening.

It ultimately resulted in a loop counter overflowing, which recomputed the same thing a billion of time (but always the same!). So the visible effect was a request would randomly take 3 min instead of 100ms.

I ended up using perf in production, which indirectly lead me to understand the data race.

I was called in to help the team because of my experience debugging the weirdest things as a platform dev.

Because of this I was exposed to so many races in Go, from my biased point of view, I want Rust everywhere instead.

But I guess I am putting myself out of a job? ;)

I think the true answer is that the moment you have to do tricky concurrency in Go, it becomes less desirable. I think that Go is still better at tricky concurrency than C, though there are some downsides too (I think it's a bit easier to sneak in a torn read issue in Go due to the presence of fat pointers and slice headers everywhere.)

Go is really good at easy concurrency tasks, like things that have almost no shared memory at all, "shared-nothing" architectures, like a typical web server. Share some resources like database handles with a sync.Pool and call it a day. Go lets you write "async" code as if it were sync with no function coloring, making it decidedly nicer than basically anything in its performance class for this use case.

Rust, on the other hand, has to contend with function coloring and a myriad of seriously hard engineering tasks to deal with async issues. Async Rust gets better every year, but personally I still (as of last month at least) think it's quite a mess. Rust is absolutely excellent for traditional concurrency, though. Anything where you would've used a mutex lock, Rust is just way better than everything else. It's beautiful.

But I struggle to be as productive in Rust as I am in Go, because Rust, the standard library, and its ecosystem gives the programmer so much to worry about. It sometimes reminds me of C++ in that regard, though it's nowhere near as extremely bad (because at least there's a coherent build system and package manager.) And frankly, a lot of software I write is just boring, and Go does fine for a lot of that. I try Rust periodically for things, and romantically it feels like it's the closest language to "the future", but I think the future might still have a place for languages like Go.

And frankly, a lot of software I write is just boring, and Go does fine for a lot of that. I try Rust periodically for things, and romantically it feels like it's the closest language to "the future", but I think the future might still have a place for languages like Go.

It's not so much about being "boring" or not; Rust does just fine at writing boring code once you get familiar with the boilerplate patterns (Real-world experience has shown that Rust is not really at a disadvantage wrt. productivity or iteration speed).

There is a case for Golang and similar languages, but it has to do with software domains where there literally is no viable alternative to GC, such as when dealing with arbitrary, "spaghetti" reference graphs. Most programs aren't going to look like that though, and starting with Rust will yield a higher quality solution overall.

Rust can yield a higher quality solution, but we can't make a perfect solution, we can only approach perfection. If we want to go further, we could introduce formally-proven code, too. Personally I'm interested in the intersection of proof assistants and Rust, like creusot-rs, and have been investigating it.

But as much as I love LARPing about correctness (believe me I do,) it's just simply the case that we won't right perfect software and it's totally OK. It's totally OK that our software will have artificial limitations, like with Go, only accepting filenames that are valid UTF-8, or taking some unnecessary performance/latency hits, or perhaps even crashing in some weird ass edge case. There are very few domains in which correctness issues can't be tolerated.

I don't deal with domains that are truly mission critical, where people could die if the code is incorrect. At worst, people could lose some money if my code is incorrect. I still would prefer not to cause that to happen, but those people are generally OK with taking that risk if it means getting features faster.

That's why Go has a future really. It's because for most software, some correctness issues are not the end of the world, and so you can rely on not fully sound approaches to finding bugs, like automated testing, race detection, and so on.

Rust can also make some types of software more productive to write, but it is unlikely to beat Go in terms of productivity when it comes to a lot of the stuff SaaS shops deal with. And boy, the software industry sure is swamped in fucking SaaS.

A lot of sass people i know are more and more choosing rust for boring code. This includes several people who said things like "go is good enough, i don't want to deal with all the rust completely".

Once your sass products get enough users, and you're dealing with millions or billions of requests per day, those rare bugs start showing up quite often... And it turns out programming towards correctness is desirable, if for no other reason than to keep pagerduty quiet. Tolerating correctness issues isn't cost-free... People having to respond during off hours costs money and stress. I think most people would rather pay the costs at dev time, when they aren't under the pressure of an incident, than during an outage.

Rust can also make some types of software more productive to write, but it is unlikely to beat Go in terms of productivity when it comes to a lot of the stuff SaaS shops deal with. And boy, the software industry sure is swamped in fucking SaaS.

I just wish Go supported parametric enums (sum types) and Option, rather than copying Hoare’s billion dollar mistake.

I ported some code to Go and rust a few years ago to try both languages out. The rust code ended up being 30% smaller because I could use an enum and a match expression. In Go I needed to make a set of types and interface{} to achieve the same thing - which was both slower and way more verbose. My rust implementation was as fast as my C implementation in 2/3rds as much code. And it was trivial to debug. My Go implementation took way more code to write - about the same amount of code as C, but it was harder to read than C and ran much slower.

For cookie cutter SAAS and prototypes, I prefer typescript. It’s fast enough for most things, and the type system is much more expressive without getting in your way. Not as convenient to deploy as go - especially on mobile. And the standard library is more like an attic. But in my opinion it’s a much better designed language.

(Real-world experience has shown that Rust is not really at a disadvantage wrt. productivity or iteration speed).

I don't believe that for a second. Even just going from Python to Go drops my productivity by maybe about 50%. Rust? Forget it.

Sure, if you have a project that demands correctness and high performance that requires tricky concurrency to achieve, something like Rust may make sense. Not for your run-of-the-mill programs though.

I believe your productivity drops as you say. I don't think it's inherent to the language though, at least not most of it. Rather it think it's a matter of familiarity and experience in each. When you're less practiced in a language, you're slower at it. I can write python fast, but im pretty slow at ruby. I've written a lot of python rust and go, and am about equally productive in them (although how that productivity is distributed through the dev cycle is different). It wasn't always this way, I was slow in each of them at first.

Rust is objectively harder to write than Go (or any other GC-language) because it exposes more concerns for the programmer to take care of. With Rust, you must always ensure your program complies with the borrow checker's rules (which is what makes Rust memory-safe) which includes having to add lifetime annotations to your variables in many cases. Go just doesn't have any of that. You could argue Rust still has an advantage in that it prevents bugs that in Go you're free to write, but then what you're claiming is that this compensates for the extra work you have to do upfront in Rust. That's a difficult position to defend, though, because an experienced Go developer probably has internalized how to avoid those bugs and the cost of preventing them can be nearly negligible. I do agree that they will still make mistakes, and those can have a cost, but they may also be quite rare, or may not matter much most of the time depending on your domain. I think that's why most people seem to agree Rust is probably only advantageous where the cost of data races in production is higher than the cognitive cost (which translates into increased effort) on the programmer.

an experienced Go developer probably has internalized how to avoid those bugs and the cost of preventing them can be nearly negligible.

And an experienced Rust developer has internalized the patterns (such as cloning or ARC) that are needed to cope with the borrow checker while writing prototype-quality, quick-iteration code. What's easier, fixing hard-to-spot bugs in the code or getting that code to compile in the first place?

Rust is objectively harder to write than Go (or any other GC-language) because it exposes more concerns for the programmer to take care of.

comparing apples to apples: Once you get a tiny bit of experience, almost all of that goes away. The common patterns and idioms in the language allow you to write whole programs without ever thinking about lifetimes or memory allocation or anything else different from the gc language case.

comparing apples to oranges: you do need to worry about those things when writing tricky memory management code that you couldn't even get from most gc lanuages... yeah then you have to worry about the things since it's a case where those things are the point.

You could argue Rust still has an advantage in that it prevents bugs that in Go you're free to write, but then what you're claiming is that this compensates for the extra work you have to do upfront in Rust.

I have evidence in the form of multiple services and programs running in prod under heavy use for years without having to revist the code to deal with bugs. Meanwhile the stuff written in go has to be touched a lot to deal with bugs. The extra couple of weeks upfront to do it in rust is mitigated after the first incident with the go code. The effort proves worthwhile after the second incident.

Also tangentially related: the cost of an incident in the form of lost business, refunds, etc is usually far higher than the cost of a couple developer weeks.

because an experienced Go developer probably has internalized how to avoid those bugs and the cost of preventing them can be nearly negligible

Some of them yes. But this is literally the same argument I'm making about rust experience meaning that you don't spend all that much extra effort up-front. Like I said, I'm about equally productive in go, python or rust.

I think that's why most people seem to agree Rust is probably only advantageous where the cost of data races in production is higher than the cognitive cost (which translates into increased effort) on the programmer.

I think people who say this haven't gotten much experience in rust. In my experience they spent a week trying to learn rust and decided to stop and compare it to their years of other languages and paradigms.

It wasn't really tricky concurrency. Somebody just made the mistake of sharing a pointer across goroutines. It was quite indirect. Boils down to a function takeing a param and holds onto it. `go` is used at some point closing over this pointer. And now we have a data race in the waiting.

Aside from the type system bypass described in the article though, this is basically no different from the status quo for virtually all languages with free threading that aren't Rust. I argue that while everyone is fallible, experienced programmers usually don't make this sort of mistake directly, because they are usually well aware that sharing pointers over a closure on another thread is a recipe for disaster. Instead, I think that most of these issues are actually involving tricky circumstances that result in this happening by accident, like accidentally re-using an err variable from the enclosing function of a closure. These sorts of bugs are really bad, because they happen in spite of everyone knowing not to inappropriately share data across goroutines, and they are easy to sneak by in code reviews unnoticed. They don't intuitively look like concurrency bugs, and sometimes they're as little as one single : away from being correct. (Which I agree is a bad place for a language design to be.)

Thankfully though, people don't just throw their hands up there; a good amount of work has gone into figuring out the kinds of mistakes that often lead to Go concurrency bugs in the real world and writing static analysis tools that can help prevent them. That work, combined with Go's builtin tools and standard library, and the memory safety of individual isolated goroutines, makes most production Go concurrency bugs fairly boring even compared to C concurrency bugs, even though they theoretically have the same basic problem where you can freely share mutable data unsafely across concurrent threads.

So yes, it is still possible to write trivial, obvious concurrency bugs. The language won't stop you. However I've used Go across almost every job I've had since like 2016 and it has been rare to come across a concurrency bug this trivial. I hope I would catch flagrantly shared mutable state across threads during code review.

Yes experienced programmers won't make the obvious mistake... until they do because the distance from the source of the bug to it's manifestation is too great to notice until it fails in production.

Look, this is pointless. I'm not learning anything new when you tell me that it can and will happen. How will it happen and how much will it happen?

Hence linking to Uber's case study on the issue. The answer? Not that much.

Uber started performing race detection in production over a 6 month period and found 2,000 different race conditions. Ouch, that sounds horrible!

But wait, we're talking about 50 million lines of Go code and 2,100 services at the time of that writing. That means they were seeing approximately 1 race condition per 25,000 lines of code and about 1 race condition per service. That actually lines up pretty well with my experiences. Although I haven't had a production outage or serious correctness issue caused by a race condition in Go, I have seen probably about one or two race conditions that made it to production per service. I reckon those codebases were likely somewhere between 10,000 and 25,000 lines of code most likely, so not so far off of the scale.

But again it doesn't always lead to a serious production outage, it's just that simple. It could be worse too (could corrupt some data and pollute your production database or something, in the worst case) but usually it's better (wonky behavior but no long-term effects, maybe the service periodically crashes but restarts, leading to some dropped requests but no long term downtime.) Uber has no doubt seen at least some Go data races that have caused actual production outages, but they've seen at least 2,000 Go data races that haven't, otherwise they would've probably been caught before the race detector caught them, Go dumps stacktraces on crash. That has to tell you something about the actual probability of causing a production outage due to a data race.

Again, you do you, but I will not be losing sleep over this. It is something to be weary of when working on Go services, but it is manageable.

Identifiable "wonky" behavior and periodic crashes seem like a very real issue to me. This wouldn't fly for any mission-critical service, it's something that demands a root cause analysis. Especially since it's hard to be sure after the fact that no data has been corrupted somehow or that security invariants have not been violated due to the "wonky" behavior.

But I struggle to be as productive

You should calculate TCO in productivity. Can you write Python/Go etc. faster? Sure! Can you operate these in production with the same TCO as Rust? Absolutely not. Most of the time the person debugging production issues and data races is different than the one who wrote the code. This gives the illusion of productivity being better with Python/Go.

After spending 20+ years around production systems both as a systems and a software engineer I think that Rust is here for reducing the TCO by moving the mental burden to write data race free software from production to development.

TCO? Tail call optimization? But that doesn't make sense in this context. hm.

It is very unfortunate that we use fixed width numbers by default in most programming languages and that common ops will silently overflow. Smarter compilers can work with richer numeric primitives and either automatically promote machine words to big numbers or throw an error on overflow.

People talk a lot about the productivity gains of ai, but fixing problems like this at the language level could have an even bigger impact on productivity, but are far less sensational. Think about how much productivity is lost due to obscure but detectable bugs like this one. I don't think rust is a good answer (it doesn't check overflow by default), but at least it points a little bit in the vaguely correct direction.

The situation with numbers in basically every widely used programming language is kind of an indictment of our industry. Silent overflow for incorrect results, no convenient facilities for units, lossy casts everywhere. It's one of those things where standing in 1975 you'd think surely we'll spend some of the next 40 years of performance gains to give ourselves nice, correct numbers to work with, but we never did.

"nice, correct numbers" end somewhere between 1/3 and sqrt(2)

so in reality, it's just "pick your own poison" to various degrees...

The square root of two is still a computable Real. We choose not to cope with that, but it's not actually impossible it was merely inconvenient. I've mentioned elsewhere that my Rust care realistic is quite happy to work with these numbers e.g. take the square root of ten, and the square root of forty, multiply them together and get the quite ordinary integer twenty.

The non computable reals are a huge problem because, as their name suggests, we can't compute them - and in the strict sense that's Almost All reals, but none of the ones you're thinking of are non-computable so you'll likely be fine.

For the merely rational numbers like a third, or sixteen hundred and five sevenths, it's even more so a matter of choosing not to address it rather than it being out of reach.

The problem with computables is that equivalence between them is only semi-decidable. (If the two numbers are different, it is decidable, but if they are not, it isn't. The problem is that you don't know if they are different a priori, so you might get lucky and find difference, but you might as well not.)

We know for sure that algebraic numbers behave nicely in terms of equivalence, and there are other, bigger number systems that are conjectured to behave nicely ( https://en.wikipedia.org/wiki/Period_(algebraic_geometry) ), but the problem with these and computers is that they are hard to represent.

Rust checks overflow by default in debug builds

I've often thought that I'd prefer it to check by default in release builds, too, but I understand that comes with a performance penalty that a lot of folks aren't happy with.

I assume this implies that common processor architectures (x86_64, aarch64) lack trap-on-overflow variants of their integer arithmetic instructions? If the explanation really is that simple, it's pretty disappointing.

Rust does have loop counter overflow.

Theoretically you can construct a loop counter that overflows, but I don't that there is any reasonable way to do it accidentally?

Within safe rust you would likely need to be using an explicit .wrapping_add() on your counter, and explicitly constructing a for loop that wasn't range-based...

It ultimately resulted in a loop counter overflowing, which recomputed the same thing a billion of time (but always the same!). So the visible effect was a request would randomly take 3 min instead of 100ms.

This means that multiple goroutines were writing to the same local variable. I've never worked on a Go team where code that is structured in such a way would be considered normal or pass code review without good justification.

What do Uber mean in that article when they say that Go programs "expose 8x more concurrency compared to Java microservices"? They're using the word concurrency as if it were a countable noun.

Wow that's a really big gotcha in go!

To be fair though, go has a big emphasis on using its communication primitives instead of directly sharing memory between goroutines [1].

[1] https://go.dev/blog/codelab-share

Real-world golang programs share memory all the time, because the "share by communicating" pattern leads to pervasive logical problems, i.e. "safe" race conditions and "safe" deadlocks.

Even if you use channels to send things between goroutines, go makes it very hard to do so safely because it doesn't have the idea of sendable types, ownership, read-only references, and so on.

For example, is the following program safe, or does it race?

    func processData(lines <-chan []byte) {
      for line := range lines {
        fmt.Printf("processing line: %v\n", line)
      }
    }

    func main() {
      lines := make(chan []byte)
      go processData(lines)

      var buf bytes.Buffer
      for range 3 {
        buf.WriteString("mock data, assume this got read into the buffer from a file or something")
        lines <- buf.Bytes()
        buf.Reset()
      }
    }

The answer is of course that it's a data race. Why?

Because `buf.Bytes()` returns the underlying memory, and then `Reset` lets you re-use the same backing memory, and so "processData" and "main" are both writing to the same data at the same time.

In rust, this would not compile because it is two mutable references to the same data, you'd either have to send ownership across the channel, or send a copy.

In go, it's confusing. If you use `bytes.Buffer.ReadBytes("\n")` you get a copy back, so you can send it. Same for `bytes.Buffer.String()`.

But if you use `bytes.Buffer.Bytes()` you get something you can't pass across a channel safely, unless you also never use that bytes.Buffer again.

Channels in rust solve this problem because rust understands "sending" and ownership. Go does not have those things, and so they just give you a new tool to shoot yourself in the foot that is slower than mutexes, and based on my experience with new gophers, also more difficult to use correctly.

That code would never pass a human pull request review. It doesn't even pass AI code review with a simple "review this code" prompt: https://chatgpt.com/share/68829f14-c004-8001-ac20-4dc1796c76...

"2. Shared buffer causes race/data reuse You're writing to buf, getting buf.Bytes(), and sending it to the channel. But buf.Bytes() returns a slice backed by the same memory, which you then Reset(). This causes line in processData to read the reset or reused buffer."

I mean, you're basically passing a pointer to another thread to processData() and then promptly trying to do stuff with the same pointer.

This is a canard.

What's happening here, as happens so often in other situations, is that a term of art was created to describe something complicated; in this case, "memory safety", to describe the property of programming languages that don't admit to memory corruption vulnerabilities, such as stack and heap overflows, use-after-frees, and type confusions. Later, people uninvolved with the popularization of the term took the term and tried to define it from first principles, arriving at a place different than the term of art. We saw the same thing happen with "zero trust networking".

The fact is that Go doesn't admit memory corruption vulnerabilities, and the way you know that is the fact that there are practically zero exploits for memory corruption vulnerabilities targeting pure Go programs, despite the popularity of the language.

Another way to reach the same conclusion is to note that this post's argument proves far too much; by the definition used by this author, most other higher-level languages (the author exempts Java, but really only Java) also fail to be memory safe.

Is Rust "safer" in some senses than Go? Almost certainly. Pure functional languages are safer still. "Safety" as a general concept in programming languages is a spectrum. But "memory safety" isn't; it's a threshold test. If you want to claim that a language is memory-unsafe, POC || GTFO.

in this case, "memory safety", to describe the property of programming languages that don't admit to memory corruption vulnerabilities, such as [..] type confusions

The fact is that Go doesn't admit memory corruption vulnerabilities

Except it does. This is exactly the example in the article. Type confusion causes it to treat an integer as a pointer & deference it. This then trivially can result in memory corruption depending on the value of the integer. In the example the value "42" is used so that it crashes with a nice segfault thanks to lower-page guarding, but that's just for ease of demonstration. There's nothing magical about the choice of 42 - it could just as easily have been any number in the valid address space.

to describe the property of programming languages that don't admit to memory corruption vulnerabilities, such as stack and heap overflows, use-after-frees, and type confusions.

And data races allow all of that. There cannot be memory-safe languages supporting multi-threading that admit data races that lead to UB. If Go does admit data races it is not memory-safe. If a program can end up in a state that the language specification does not recognize (such as termination by SIGSEGV), it’s not memory safe. This is the only reasonable definition of memory safety.

If that were the case, you'd be able to support the argument with evidence.

You mean like the program in the article where code that never dereferences a non-pointer causes the runtime to dereference a non-pointer? That seems like evidence to me.

An exploit against a real Go program that relies on memory corruption.

It should be possible to construct an exploit for such programs. But even for truly unsafe languages, vulnerabilities just from data races are very rare, because they are much harder to exploit.

You could argue Go is safe from memory vulnerabilities, and that'll be 99% correct (we can't know what will happen if some very strong organization (e.g. a nation-state actor) will heavily invest in exploiting some Go program), but it still isn't memory safe, as per the definition in Wikipedia:

Memory safety is the state of being protected from various software bugs and security vulnerabilities when dealing with memory access, such as buffer overflows and dangling pointers.

There's enormous incentive to construct those exploits. Why don't they exist?

This seems to be operating on the premise that everyone knows of every exploit that exists. There's also an enormous incentive to hide working exploits.

I'm not sure that's correct.

Yes, this is an enormous effort to construct exploits, but constructing exploits for C/C++ code is much much easier and gives not less, or even more, benefit. Therefore it makes sense the efforts are focused on that.

If/when most C/C++ code in the world will be gone, I assume we'll see more exploits of Go code.

You’re a cryptography person. So you know that most theoretically interesting cryptography vulnerabilities, even the ones that are exploitable in PoCs, are too obscure and/or difficult to get used by actual attackers. Same goes for hardware vulnerabilities. Rowhammer and speculative execution attacks are often shown to be able to corrupt and leak memory, respectively, but AFAIK there are no famous cases of them actually being used to attack someone. Partly because they’re fiddly; partly out of habit. Partly because if you’re in a position to satisfy the requirements for those attacks – you have copies of all the relevant binaries so you know the memory layout of code and data, you have some kind of sandboxed arbitrary code execution to launch the attack from – then you’re often able to find better vulnerabilities elsewhere. And the same is also true for certain types of software vulnerabilities…

Honestly, forget about Go: when was the last time you heard of a modern application backend being exploited through memory corruption, in any language? I know that Google and Meta and the like use a good amount of C++ on the server, as do many smaller companies. That C++ code may skew ‘modern’ and safer, but you could say the same about newly-developed client-side C++ code that’s constantly getting exploited. So where are the server-side attacks? Part of the answer is probably that they exist, but I don’t know about them because they haven’t been disclosed. Unlike client-side attacks, server-side attacks usually target a single entity who has little incentive to publish deep dives into how they were attacked. That especially applies to larger companies, which tend to use more C++. But we do sometimes see those deep dives written anyway, and the vulnerabilities described usually aren’t memory safety related. So I think there is also a gap in actual exploitation. Which probably has a number of causes, but I’d guess they include attackers (1) usually not having ready access to binaries, (2) not having an equivalent to the browser as a powerful launching point for exploits, and (3) not having access to as much memory-unsafe code as on the client side.

This is relevant to Go because of course Go is usually used on the server side. There is some use of Go on the client side, but I can’t think offhand of a single example of it being used in the type of consumer OS or client-side application that typically gets attacked.

Meanwhile, Go is of course much safer than C++. To make exploitation possible in Go, not only do you need a race condition (which are rarely targeted by exploits in any language), you also need a very specific code pattern. I’m not sure exactly how specific. I know how a stereotypical example of an interface pointer/viable mismatch works. But are there other options? I hear that maps are also thread-unsafe in general? I’d need to dig into the implementation to see how likely that is to be exploitable.

Regardless, the potential exists. If memory safety is a “threshold test” as you say, then Go is not memory-safe.

I agree though that the point would best be proven with a PoC of exploiting a real Go program. As someone with experience writing exploits, I think I could probably locate a vulnerability and create an exploit, if I had a few months to work on it. But for now I have employment and my free time is taken up by other things.

https://github.com/golang/go/issues/34902

https://www.cloudfoundry.org/blog/cve-2020-15586/

I don’t see any evidence that anyone wrote an RCE exploit for this, but I also don’t see any evidence of anyone even trying to rule it out.

What about this particular bug do you think makes it likely to be exploitable? I'm not asking you to write an RCE POC, just to tell a story of the sequence of events involving this bug that results in attacker-controlled code. What does the attacker control here, and how do they use that control to divert execution?

As a general heuristic, a corrupted data structure in a network server results in RCE. This is common in languages like C and C++.

On first glance, it looks like the bug can (at least) result in the server accessing a slice object where the various fields don’t all come from the same place. So the target server can end up accessing some object out of bounds (or as the wrong type or both), which can easily end up writing some data (possibly attacker controlled) to an inappropriate place. In standard attack, the attacker might try to modify the stack or a function pointer to set up a ROP chain or something similar, which is close enough to arbitrarily code to eventually either corrupt something to directly escalate privileges or to do appropriate syscalls to actually execute code.

Hide the same program into some dependency of a dependency and you have a nice little security vulnerability in your prod app. It's actually very easy to hide such a vulnerability as an innocent bug.

If you're stipulating deliberately inserted vulnerabilities then there are much easier ways, e.g., with a plausibly-deniable logic bug in code that calls os/exec or reflect (both of which can execute arbitrary code by design).

If you see `exec`, that's an obvious point where you want to pay extra attention.

Compare to an innocent looking map operation, and it's not even in the same league.

I think your security background is coloring your perception of the term memory safety. Specifically the requirement that the various issues lead to exploitation. These issues can lead to many other issues that are not vulnerability in the security sense, e.g. data corruption, incorrect (but not insecure) behavior, performance issues, and more. I don't think any of those were ever dismissed or excluded from memory safety discussion. Infosec circles tend to evaluate most ideas in the context of (anti)exploitation, and the rest of programming tends to focus on what the cool kids argue (that is they often weigh security concerns higher than other issues as well), so the other problems caused by double-free or buffer overruns (etc) just may not have been given as much weight in your mind.

"Memory safety" is a security term, not a PLT term.

That's a statement without source and even if it was widely accepted as true it doesn't imply the fact that something needs to be exploitable to be considered a security issue. We are full of CVEs without a known way to be exploited.

This is a good post and I agree with it in full, but I just wanted to point out that (safe) Rust is safer from data races than, say, Haskell due to the properties of an affine type system.

Haskell in general is a much safer than Rust thanks to its more robust type system (which also forms the basis of its metaprogramming facilities), monads being much louder than unsafe blocks, etc. But data races and deadlocks are one of the few things Rust has over it. There are some pure functional languages that are dependently typed like Idris, and thus far safer than Rust, but they're in the minority and I've yet to find anybody using them industrially. Also Fortnite's Verse thing? I don't know how pure that language is though.

If you want to claim that a language is memory-unsafe, POC || GTFO.

There's a POC right in the post, demonstrating type confusion due to a torn read of a fat pointer. I think it could have just as easily been an out-of-bounds write via a torn read of a slice. I don't see how you can seriously call this memory safe, even by a conservative definition.

Did you mean POC against a real program? Is that your bar?

You need a non-contrived example of a memory-corrupting data race that gives attackers the ability to control memory, through type confusion or a memory lifecycle bug or something like it. You don't have to write the exploit but you have to be able to tell the story of how the exploit would actually work --- "I ran this code and it segfaulted" is not enough. It isn't even enough for C code!

The post is a demonstration that a class of problems: causing Go to treat a integer field as a pointer and access the memory behind that pointer without using any of Go's documented "unsafe.Pointer" (or other documented as unsafe operations).

We're talking about programming languages being memory safe (like fly.io does on it's security page [1]), not about other specific applications.

It may be helpful to think of this as talking about the security of the programming language implementation. We're talking about inputs to that implementation that are considered valid and not using "unsafe" marked bits (though I do note that the Go project itself isn't very clear on if they claim to be memory-safe). Then we want to evaluate whether the programming language implementation fulfills what people think it fulfills; ie: "being a memory safe programming language" by producing programs under some constraints (ie: no unsafe) that are themselves memory-safe.

The example we see in the OP is demonstrating a break in the expectations for the behavior of the programming language implementation if we expected the programming language implementation to produce programs that are memory safe (again under some conditions of not using "unsafe" bits).

[1]: https://fly.io/docs/security/security-at-fly-io/#application...

The thread you're commenting has already discussed everything this comment says.

If you've got concerns about our security page, I think you should first take them to the ISRG Prossimo project.

https://www.memorysafety.org/docs/memory-safety/

In this thread I linked the fly.io security page because it helps us establish that one can talk about _languages_ as being memory safe specifically, which is something it seems you're rejecting as a concept in the parent and other comments.

(In a separate comment about "what do people claim about Go anyhow", I linked the memorysafety.org page, but I did not expect it to help in getting you to the understanding that we can evaluate programming languages as being memory safe or not, where something from the company where someone was a founder seemed more likely to get a person to reconsider the framing of what we're examining)

Huh? No, I'm not. Go is a memory-safe programming language, like Java before it, like Python, Ruby, Javascript, and of course Rust.

So you're saying nobody cares about actual memory safety in concurrent code? Then why did the Swift folks bother to finally make the language memory-safe (just as safe as Rust) for concurrent code? Heck why did the Java folks bother to define their safe concurrency/memory model to begin with? They could have done it the Golang way and not cared about the issue.

The fact is that Go doesn't admit memory corruption vulnerabilities, and the way you know that is the fact that there are practically zero exploits for memory corruption vulnerabilities targeting pure Go programs, despite the popularity of the language.

Another way to word it: If "Go is memory unsafe" is such a revelation after its been around for 13 years, it's more likely that such a statement is somehow wrong than that nobody's picked up on such a supposedly impactful safety issue in all this time.

As such, the burden of proof that addresses why nobody's ran into any serious safety issues in the last 13 years is on the OP. It's not enough to show some theoretical program that exhibits the issue, clearly that is not enough to cause real problems.

There's no "revelation" here, it's always been well known among experts that Go is not fully memory safe for concurrent code, same for previous versions of Swift. OP has simply spelled out the argument clearly and made it easier to understand for average developers.

It's made what would be a valid point using misleading terminology and framing that suggests these are security issues, which they simply are not.

"One could easily turn this example into a function that casts an integer to a pointer, and then cause arbitrary memory corruption."

No, one couldn't! One has contrived a program that hardcodes precisely the condition one wants to achieve. In doing so, one hasn't even demonstrated even one of the two predicates for a memory corruption vulnerability (attacker control of the data, and attacker ability to place controlled data somewhere advantageous to the attacker).

What the author is doing is demonstrating correctness advantages of Rust using inappropriate security framing.

misleading terminology and framing that suggests these are security issues

Could you quote where exactly OP has misleadingly "suggested" that these concerns lead to security issues in the typical case?

attacker control of the data, and attacker ability to place controlled data somewhere advantageous to the attacker

Under this definition the Rowhammer problem with hardware DRAM does not qualify as a genuine security concern since it inherently relies on fiddly non-determinism that cannot possibly be "controlled" by any attacker. (The problem with possible torn writes in concurrent Go code is quite similar in spirit; it's understood that an actually observed torn write might only occur rarely.) Needless to say there is a fairly strong case for addressing these problems anyway, as a matter of defence in depth.

correctness advantages of Rust

Memory safety in OP's sense is not exclusive to Rust. Swift has it. Even Java/C# cannot access arbitrary memory as a result of torn writes. It would be more accurate to say that OP has identified a correctness issue that's apparently exclusive to Go.

I quoted directly from the article.

To use your definition, that quote is clearly making a point about correctness, not necessarily about real-world security.

Nope. You can have programs without undefined behavior and still not have thread safety. In .NET, for example, writes to variables that are wider then the machine width or not aligned properly, are not guaranteed to be atomic. So if you assign some value to an Int128 variable, it will not be updated atomically - how could it, that is just beyond the capabilities of the processor - and therefore a different thread can observe a state where only half of the variable has been updated. No undefined behavior here but also sharing this variable between threads is not thread safe. And having the language synchronize all such writes - just in case some other thread might want tot look at it - is a performance disaster. And disallowing anything that might be a potential thread safety issue will give you a pretty limited language.

The statement "there is no memory safety without thread safety" does not suggest that memory safety is sufficient to provide thread safety. Instead, it's just saying that if you want thread safety, then memory safety is a requirement.

Instead, it's just saying that if you want thread safety, then memory safety is a requirement.

It's saying the opposite – that if you want memory safety, thread safety is a requirement – and Java and C# refute it.

See the article's comments on Java, which is "thread safe" in the sense of preventing undefined behavior but not in the sense of preventing data-race-related logic bugs. .NET is precisely analogous in this respect.

I can buy that claim for the .NET CLR but I've never seen it nailed down properly the way Java did which gives me pause.

I worry about the Win95-era "Microsoft Pragmatism" at work and a concrete example which comes to mind is nullability. In the nice modern software I often work on I can say some function takes a string and in that program C# will tell me that's not allowed to be null, it has to be an actual string - a significant engineering benefit. But, the CLR does not enforce such rules, so that function may still receive a null instead e.g. if called by some ten year old VB.NET code which has no idea about "nullability" and so just fills out a null for that parameter anyway.

Of course the CLR memory model might really be set in stone and 100% proof against such problems, but I haven't seen anything to reassure me as I did for Java and I fear that if it were convenient for Windows to not quite do that work they would say eh, good enough.

There's a documented memory model (https://github.com/dotnet/runtime/blob/main/docs/design/spec...), does that not address this concern?

This is false as a generality.

A memory safe, managed language doesn't become unsafe just because you have a race condition in a program.

Like, say, reading and writing several related shared variables without a mutex.

Say that the language ensures that the reads and writes themselves of these word-sized variables are safe without any lock, and that memory operations and reclamation of memory are thread safe: there are no low-level pointers (or else only as an escape hatch that the program isn't using).

The rest is your bug; the variable values coming out of sync with each other, not maintaining the invariant among their values.

It could be the case that a thread-unsafe program breaks a managed run-time, but not an unvarnished truth.

A managed run-time could be built on the assumption that the program will not create two or more threads such that those threads will invoke concurrent operations on the same objects. E.g. a managed run time that needs a global interpreter lock, but which is missing.

If the variables are word-sized, sure. But what if they are larger? Now a race condition between one thread writing and another thread reading or writing a variable is a memory safety issue.

Now a race condition between one thread writing and another thread reading or writing a variable is a memory safety issue.

No it isn't, because the torn write cannot have arbitrary effects that potentially break the program. It only becomes such if you rely on such a variable to establish an invariant about memory that's broken if a torn write occurs (such as by encoding a ptr+len in it), which is just silly. Don't do that!

The point being made is sound, but I can never escape the feeling that most concurrency discussion in programming language theory is ignoring the elephant in the room. The concurrency bugs that matter in most apps are all happening inside the database due to lack of proper locking, transactions or transactional isolation. PL theory ignores this and so things like Rust's approach to race freedom ends up not mattering much outside of places like kernels. A Rust app can avoid use of unsafe entirely and still be riddled with race conditions because all the data that matters is in an RDBMS and someone forgot a FOR UPDATE in their SELECT clause.

I agree with the author's claim that you need thread safety for memory safety.

But I don't agree with:

I will argue that this distinction isn’t all that useful, and that the actual property we want our programs to have is absence of Undefined Behavior.

There is plenty of undefined behavior that can't lead to violating memory safety. For example, in many languages, argument evaluation order is undefined. If you have some code like:

    foo(print(1), print(2));

In some languages, it's undefined as to whether "1" is printed before "2" or vice versa. But there's no way to violate memory safety with this.

I think the only term the author needs here is "memory safety", and they correctly observe that if the language has threading, then you need a memory model that ensures that threads can't break your memory safety.

Go lacks that. It seems to be a rare problem in practice, but if you want guarantees, Go doesn't give you them. In return, I guess it gives you slightly faster execution speed for writes that it allows to potentially be torn.

That's "unspecified" not "undefined". "Undefined behavior" literally means "anything goes", so any program that invokes it is broken by definition.

That is not true, that is a very specific definition of UB which C developers (among others) favor. That doesn't mean that another language can't say "this is undefined behavior" without all the baggage that accompanies the term in C.

It's literally how the term "UB" is defined, and understood by experts. Why would anyone want to say "undefined" when they really mean "unspecified"? That's just confusing.

No, it's how one very specific community of experts understands it. It is not some kind of universal law of definition that it must mean that always and everywhere. As far as what is confusing, that is a matter of perspective. I think it is confusing (to put it mildly) that the C community has chosen to use "undefined behavior" to mean "it must never happen, and anything goes if it does". That is extremely counterintuitive, and only makes sense to those who live and breathe that world. So if the standard is to be "avoiding confusion", then we better change the definition used by the C community ASAP.

It is also not what the C community has chosen. It is what was imposed on us by certain optimizing compilers that used the interpretation that gave them maximum freedom to excel in benchmarks, and it was then endorsed by C++. The C definition is that "undefined behavior" can have arbitrary concrete behavior, not that a compiler can assume it does not happen. (that form semantic people prefer the former because it makes their life easier did not help)

The C definition is that "undefined behavior" can have arbitrary concrete behavior, not that a compiler can assume it does not happen.

What is the difference between those? How does a compiler that assumes UB never happens violate the requirement that UB can have arbitrary concrete behavior? If we look at a simple example like optimizing "x + y > x" (signed arithmetic, y known to be positive) to "true" -- that will lead to some arbitrary concrete behavior of the program, so it seems covered by the definition.

I assume that what the original C authors meant was closer to "on signed integer overflow, non-deterministically pick some result from the following set", but that's not what they wrote in the standard... if you want to specify that something is non-deterministic, you need to spell out exactly what the set of possible choices are. Maybe for singed integer overflow one could infer this (though it really should be made explicit IMO), but C also says that the program has UB "by default" if it runs into a case not described by the standard, and there's just no way to infer a set of choices from that as far as I can see.

"arbitrary concrete behavior" means that at this point anything can happen on the real machine. This implies that everything before this point has to behave according to the specification. "is impossible" is stronger, as the whole program could behave erratically. But having partial correctness is important in a lot of scenarios and this is why we want to have it and in "UB" it is the former and not "impossible".

In the ISO C standard, we use "unspecified" for a non-deterministic choice among clearly specified alternatives. So this is well understood.

"Undefined behavior" is not a meaningless made up term that you can redefine at will.

The word "undefined" has a clear meaning: there is no behavior defined at all for what a given piece of code will do, meaning it can literally do anything. If the language spec defines the possible behaviors you can expect (even if the behavior can vary between implementations), then by definition it's not undefined.

"Undefined behavior" is not a meaningless made up term that you can redefine at will.

Sure, I agree with that.

The word "undefined" has a clear meaning: there is no behavior defined at all for what a given piece of code will do...

That is true, but...

...meaning it can literally do anything.

This is not at all true! That is a different (but closely related) matter, which is "what is to be done about undefined behavior". Which is certainly something one has to take a stance on when working to a language spec that has undefined behavior, but that does not mean that "undefined" automatically means your preferred interpretation of how to handle undefined behavior.

The evaluation order is _unspecified_, not undefined behaviour.

Interestingly, at least in C++, this was changed in the recent past. It used to be that evaluation of arguments was not sequenced at all and if any evaluation touched the same variable, and at least one was a write, it was UB.

It was changed as part of the C++11 memory model and now, as you said, there is a sequenced-before order, it is just unspecified which one it is.

I don't know much about C, but I believe it was similarly changed in C11.

Yes, but that's just a subset of expressions where unspecified sequencing applied. For instance, the example with two `print()` as parameters would have a sequence point (in pre-C++11 terminology) separating any reads/writes inside the `print` due to the function calls. It would never be UB even though the order in which the prints are called is still unspecified.

IIRC the point was that there was no sequence point between argument evaluation, so for example f(++i, ++i) was UB. Or maybe it was only for builtin operators?

Cppreference is not authoritative[1], but seems to support my recollection. In fact it states that the f(++i, ++i) was UB till C++17.

[1] https://en.cppreference.com/w/cpp/language/eval_order.html, Pre C++11 Ordering Rules, point (2).

`f(++i, ++i)` is/was indeed UB, but the example in munificent's comment was `foo(print(1), print(2))` which as far as I know is not even if both `print` calls read/write the same memory.

Sure, prior to the C++ 11 memory model there just isn't a memory ordering model in C++ and all programs in either C or C++ which would need ordering for correctness did not have any defined behaviour in the language standard.

This is very amusing because that means in terms of the language standard Windows and Linux, which both significantly pre-date C++ 11 and thus its memory model, were technically relying on Undefined Behaviour. Of course, as operating systems they're already off piste because they're full of raw assembly and so on.

Linux has its own ordering model as a result, pre-dating the C++ 11 model. Linus is writing software for multi-processor computers more than a decade before the C++ 11 model so obviously he can't wait around for that.

[Edit: Corrected Linux -> Linux when talking about the man]

The sad thing is that most languages with threads have a default of global variables and unrestricted shared memory access. This is the source of the vast majority of data corruption and races. Processes are generally a better concurrency model than threads, but they are unfortunately too heavyweight for many use cases. If we defaulted to message passing all required data to each thread (either by always copying or tracking ownership to elide unnecessary copying), most of these kinds of problems would go away.

In the meantime, we thankfully have agency and are free to choose not to use global variables and shared memory even if the platform offers them to us.

Message passing can easily lead to more logical errors (such as race conditions and/or deadlocks) than sharing memory directly with properly synchronized access. It's not a silver bullet.

The sad thing is that most languages with threads have a default of global variables and unrestricted shared memory access. This is the source of the vast majority of data corruption and races. Processes are generally a better concurrency model than threads

Modern languages have the option of representing thread-safety in the type system, e.g. what Rust does, where working with threads is a dream (especially when you get to use structured concurrency via thread::scope).

People tend to forget that Rust's original goal was not "let's make a memory-safe systems language", it was "let's make a thread-safe systems language", and memory safety just came along for the ride.

Originally Rust is something altogether different. Graydon has written about that extensively. Graydon wanted tail calls, reflection, more "natural" arithmetic with Python style automatic big numbers, decimal for financial work and so on.

The Rust we have from 1.0 onwards is not what Graydon wanted at all. Would Graydon's language have been broadly popular? Probably not, we'll never know.

While at it, I suppose it's straightforward to implement arbitrary-precision integers and decimals in today's Rust; there are several crates for that. There's also a `tailcall` crate that apparently implements TCO [1].

[1]: https://docs.rs/tailcall/latest/tailcall/

Even in pre-1.0 Rust, concurrency was a primary goal; there's a reason that Graydon listed Newsqueak, Alef, Limbo, and Erlang in the long list of influences for proto-Rust.

And yet it ignored the primary lesson of Erlang: no shared memory access whatsoever, which is what makes it so robust.

Because the other teams members (IIRC brson and pcwalton) wanted Rust to be as performant as C++, which means you must have a way to have shared memory.

Which was the ultimate failure of Rust, because as Pony benchmarks have shown, you get safety and speed by proper security and architecture. Rust just survived by lying about the its safeties.

What kills performance are not memory copies, but locks. Parallel nonblocking IO and a non POSIX stdlib will bring you far away from C++ or Rust performance.

What kills performance are not memory copies, but locks.

I'm pretty sure if every thread executing an LLM model had to have its own copy that that would murder performance more than any lock does, and it won't even be close.

It's cheaper to copy than to lock when the data is small, but that does not scale and it also ignores things like reader/writer locks where the data is primarily read-only, at least during the concurrent stage. Or where the work can be safely chunked up such that writes don't ever overlap which is very common in graphics

Oh yeah, the “ultimate failure of Rust”, and tell me how industrially successful Pony has been compared to Rust?

(Don't get me wrong, I liked the idea behind Pony for backend code, it's much saner than Go for the same target space. But it failed to capture that market, because Go was already there. And it was never a competitor to Rust because this model is only viable for web back end tasks, not for general computing).

Every time this conversation comes up, I'm reminded of my team at Dropbox, where it was a rite of passage for new engineers to introduce a segfault in our Go server by not synchronizing writes to a data structure.

Swift has (had?) the same issue and I had to write a program to illustrate that Swift is (was?) perfectly happy to segfault under shared access to data structures.

Go has never been memory-safe (in the Rust and Java sense) and it's wild to me that it got branded as such.

Swift is in the process of fixing this, but it’s a slow and painful transition; there’s an awful lot of unsafe code in the wild that wasn’t unsafe until recently.

Swift 6 is only painful if you wrote a ton of terrible Swift 5, and even then Swift 5 has had modes where you could gracefully adopt the Swift 6 safety mechanisms for a long time (years?)

~130k LoC Swift app was converted from 5 -> 6 for us in about 3 days.

It is still incomplete and a mess. I don't think they thought through the actual main cases Swift is used for (ios apps), and built a hypothetical generic way which is failing on most clients. Hence lots of workarounds, and ways to get around it (The actor system). The isolated/nonisolated types are a bit contrived and causing real productivity loss, when the old way was really just 'everything ui in main thread, everything that takes time, use a dispatch queue, and call main when done'.

Swift is strating to look more like old java beans. (if you are old enough to remember this, most swift developers are too young). Doing some of the same mistakes.

Anways https://forums.swift.org/t/has-swifts-concurrency-model-gone... Common problems all devs face: https://www.massicotte.org/problematic-patterns

Anyways, they are trying to reinvent 'safe concurrency' while almost throwing the baby with the bathwater, and making swift even more complex and harder to get into.

There is ways to go. For simple apps, the new concurrency is easy to adopt. But for anything that is less than trivial, it becomes a lot of work, to the point that it might not make it worth it.

Mostly because it was a remarkable improvement over what came before (and what came before was hilariously fragile).

Only for those not paying attention outside mainstream, or too young to remember former languages.

or too young to remember former languages.

Do you have any good examples? Not trying to argue, just genuinely curious as someone who hasn't been in this field for decades.

Safety isn't binary, so your comment makes no sense.

I’d argue that unsafety is binary. If a normal eng doing normal things can break it without going out of their way to deliberately fool the compiler or runtime, I’d call it unsafe.

By that definition Rust also counts as unsafe. Even managed languages like C# and Java would be unsafe.

"Memory safety" has a specific, binary definition.

It is kind of wild that for a 21st century programming language, the amount of stuff in Go that should have been but never was, but hey Docker and Kubernetes.

On the flip side, what would be the point? There are already a million other languages that have everything and the kitchen sink.

Not going down the same road is the only reason it didn't end up on the pile of obscure languages nobody uses.

The only reason it didn't end on pile of obscure languages nobody uses, it called Google, followed by luck with Docker and Kubernetes adoption on the market, after they decided to rewrite from Python and Java respectively into Go, after Go heads joined their teams.

Case in point, Limbo and Oberon-2, the languages that influenced its design, and authors were involved with.

The only reason it didn't end on pile of obscure languages nobody uses, it called Google

Dart ended up on the pile of languages nobody uses. And Carbon? What's Carbon? Exactly!

Case in point, Limbo and Oberon-2, the languages that influenced its design

Agreed. Limbo and Oberon-2, as primitive as they may look now, had the kitchen sinks of their time. Why wouldn't they have ended up on the pile of languages nobody uses?

People love to bring those as counter examples, without actually knowing a single fact about them.

Dart was a victim of internal politics between the Chrome team, Dart team, AdWords moving away from GWT wanting AngularDart (see Angular documentary), and the Web in general.

Had Chrome team kept pushing DartVM, it might have been quite different story.

Carbon, good example of failure to actually know what the team purposes are. It is officially a research project for Google themselves, where the team is the first to advise using Rust or another MSL.

One just needs to actually spend like a couple of minutes on their wiki, but I guess that is asking too much on modern times.

Limbo and Oberon-2 were definitely not kitchen sinks of their time, their failure was that neither Bell Labs in 1996, nor ETHZ in 1992, were that relevant for the programming language community in the industry.

Had Chrome team kept pushing DartVM, it might have been quite different story.

Trouble with that line of thinking is that Google never pushed Go either. It didn't even bother to use it internally (outside from the occasional side project here and there). Google paid some salaries. I'll give you that. But it has paid salaries for a lot of different languages. That is not some kind of secret sauce.

It is officially a research project for Google themselves

It's not just a research project. It is officially "not ready for use", but its roadmap has a clear "ready for use" plan in the coming months. Rust was also "not ready for use" when it hit the streets, it officially being a Mozilla research project, but every second discussion on HN was about it and what is to come. And that was without Google backing. If what you say is true, why isn't Carbon being shouted from every rooftop right now?

I know you're struggling to grasp at straws here, but let's just be honest for a moment: If it hasn't caught attention already, it isn't going to. Just another language to add to the pile.

I guess Kubernetes doesn't count as Google pushing it then.

It is officially "not ready to use", it isn't a strawman as people keep complaining about nothing.

The strength of Go is not the language. It's that the libraries you need for web back-end stuff are written, maintained, and used in production by Google. All the obscure cases get exercised in production due to sheer volume of internal usage.

At one time, Go maps were not thread-safe. Was that fixed?

I don't think that's the (only) reason Go became popular. The huge thing about Go is the runtime: it's the only language runtime available today, at least in any language with a large org behind it, that offers (a) GC, (b) fast start-up time, (c) static types, (d) fast execution, and (e) multi-threading.

This is a killer combination for any team looking to write code for auto-scalable microservices, to run for example on Kubernetes. Java is not great in this niche because of its slow startup time, relatively large memory overhead, and the need for warm-up before code actually starts executing fast (so scaling up and down has a very large cost for Java services). .NET has similar problems, and also a huge container size. Python is far too slow, and not typed. TypeScript is single threaded, and still has a pretty hefty runtime. OCaml doesn't have any large org behind it, is quite obscure syntax, and was still single-threaded at the time Kubernetes started. Haskell has similar issues, and is also large and slow starting. Rust, C++, C all require manual memory management.

So, it's no surprise that Go was used for Kubernetes services themselves, and it's no surprise that people designing for Kubernetes mostly chose to write their new stuff in Go. Go the language, with its antiquated design, is actually quite secondary to all of that. But Go's runtime is completely unmatched in this space.

F# does that too.

.NET has similar problems

s/has/had/

https://blog.washi.dev/posts/tinysharp/

The issue is that some people still fighting against the concepts ML family languages (primarily SML) introduced. Go implemented go routines and channels from CSP (https://en.wikipedia.org/wiki/Communicating_sequential_proce...) but dragged a lot on influence from C (understandable) into the language.

I think Rust opted for the best combinations (some CSP, a lot of ML and a bit of C++).

Kubernetes services are one of the places where you don't care about startup time. Likewise for Docker itself. These are the things that do the scaling, normally.

Go is not particularly fast. People often see that Java gets faster as it runs and thinks, oh, it must be slow at the start then. But when you compare like with like, Go ends up being stuck at the bottom of the curve that Java then sharply climbs. The difference in GC quality is big, or at least, used to be? The only place where you really notice the difference is command line tools, and Java has GraalVM for that.

Kubernetes services are one of the places where you don't care about startup time.

There are some kubernetes services that scale up and down. And even for those that don't normally, if they have some kind of failure, the difference between taking a millisecond to get back up and taking a second can actually matter for a web host.

Go is not particularly fast. People often see that Java gets faster as it runs and thinks, oh, it must be slow at the start then. But when you compare like with like, Go ends up being stuck at the bottom of the curve that Java then sharply climbs.

Go starts up much faster than Java. And Go code runs measurably faster than interpreted Java code, even though it's slower than the JITed code you'll eventually have if your JVM runs long enpigh. But un-JITed Java code is very slow, more comparable to Python than JITed Java or with Go . This has nothing to do with the GC - where I do agree Go is mediocre at best.

I wouldn't call the Go GC mediocre, it's one of the few fully concurrent GC's in common use. It probably has significantly lower memory demand than Java/NET for comparable workloads.

Well, that and the slight fact that it bears Google's brand name.

I personally appreciate Go as a research experiment. Plenty of very interesting ideas, just as, for instance, Haskell. I don't particularly like it as a development language, but I can understand why some people do.

Plenty of very interesting ideas

Is there? When you get down to it, it is really just a faster Python. Which is exactly what it was said to be when it was released. Their goal was to create a "dynamically-typed" language that was more performant. It is likely that it wouldn't have had a static type system at all if they figured out how to achieve on the performance end without needing types.

You can tell who is clueless when you hear someone say its type system is lacking. I mean, technically it is, but it is supposed to be. Like saying Javascript or Ruby's type system is lacking.

Two examples of interesting ideas:

- using zero values as an optimization mechanism;

- (non-)pointers and passing self by copy.

I mean, I hate both mechanisms, but intellectually, I find them quite interesting.

Also, I'd not classify it as a faster Python. It's more of a cousin of Obj-C if the authors of Obj-C had fallen in love of Erlang instead of Smalltalk.

Having the weight of Google behind it is the primary reason it didn't end up on the pile of obscure languages nobody uses.

Oh...? Dart never gained much steam. And let's not forget about Carbon! Can you name even just one person who has tried Carbon? Have more than a handful of people even heard of Carbon?

I will grant you that Carbon is still in its infancy, but when Rust was in the same youthful stage we never heard an end to all the people playing with it. You, even if not tried it yourself, definitely knew about it.

You've made up a fun idea, but reality doesn't support it. Google has not shown its weight carries anything. They have really struggled to get any for-profit business units off the ground since they gained the weight, never mind their hobbies! If anything, Google is detrimental to a project.

Already explained in another thread, learn the politics of Dart, and Carbon is still on the drawing board.

Already...? Said "explanation" was posted over an hour after the comment replied to here.

If only Google put their weight into a watch, maybe you'd have one?

Oh wait. They did! Google can't successfully turn their weight into much of anything. Go's success, if we can call it that, clearly happened in spite of Google.

Googlers aren’t expected to wear a Google-branded watch at work. They are expected to write go. Having an entire Google’s worth of programmers using your programming language isn’t exactly a minor influence.

So failures are some deep valid reasons whereas success is developers don't know any better language.

Many don't, otherwise they would not do stuff like using Python for performance workloads.

Lets also not forget Rob Pike famous quote regarding simple minds, as target audience.

As for Go, Kubernetes made it unavoidable, it is like UNIX for C, Web for JavaScript, and so forth.

I am curious. Generally basic structures like map are not thread safe and care has to be taken while modifying it. This is pretty well documented in go spec. In your case in dropbox, what was essentially going on?

In Java, there are separate synchronized collections, because acquiring a lock takes time. Normally one uses thread-unsafe collections. Java also gives a very ergonomic way to run any fragment under a lock (the `synchronized` operator).

Rust avoids all this entirely, by using its type system.

I think the surprise here is that failing to synchronize writes leads to a SEGFAULT, not a panic or an error. This is the point GP was making, that Go is not fully memory safe in the presence of unsynchronized concurrent writes. By contrast, in Java or C#, unsynchronized writes will either throw an exception (if you're lucky and they get detected) or let the program continue with some unexpected values (possibly ones that violate some invariants). Getting a SEGFAULT can only happen if you're explicitly using native code, raw memory access APIs, or found a bug in the runtime.

Segfault sounds better than running with inconsistent data.

The inconsistent data thing could happen in Go too. A segfault is not guaranteed, it’s just one of the more likely possibilities.

A segfault is not guaranteed, it’s just one of the more likely possibilities.

Is it? It will depend on the code, but my gut feeling is that you typically would get a few (if not lot of) unnoticed non-segfaulting issues before you get the segfaulting one that tells you straight in your face that you have a problem.

It probably depends on how exactly the corruption happens. If you overwrite a pointer with an integer value, then the integer is statistically unlikely to correspond to a valid memory address. On the other hand, if you overwrite a pointer with a pointer, or an integer with an integer, all bets are off.

If you overwrite a pointer with an integer value, then the integer is statistically unlikely to correspond to a valid memory address

On 64-bit systems, and even then, it depends on the system’s memory layout (I think most integer values in programs are < 2³²)

Well it depends on what we mean by 'inconsistent'.

In C# For example, if a structure is over CPU arch Word size (i.e. 32 or 64 bits) then you could have a torn read if it's being written. However object refs themselves are always word size so you'll never have a torn pointer read on those.

However, in either case there is still a need in multithreaded environments to remember the CPU's memory ordering rules and put proper fences (or, to be safe, locks, since memory barrier rules are different between ARM and x86 for example).

But that second bit is a fairly hard problem to solve for without having the right type of modelling around your compiler.

Crashing on shared access is the safe thing to do

An intentional exit by a runtime is a safe crash. A segfault is not, and is here a clear sign that memory safety has been violated.

a segfault is completely unintentional. Had the kernel been older it could be used to execute code.

a segfault is completely unintentional

Usually, but not always! https://jcdav.is/2015/10/06/SIGSEGV-as-control-flow/

Faulted trying to access 0x10 - the offset in the string we were trying to read from :)

Is guaranteed that every offset you can try to read is guaranteed to create a segfault?

Right, the issue here is that the "Rust and Java sense" of memory safety is not the actual meaning of the term. People talk as if "memory safety" was a PLT axiom. It's not; it's a software security term of art.

This is just two groups of people talking past each other.

It's not as if Go programmers are unaware of the distinction you're talking about. It's literally the premise of the language; it's the basis for "share by communicating, don't communicate by sharing". Obviously, that didn't work out, and modern Go does a lot of sharing and needs a lot of synchronization. But: everybody understands that.

People talk as if "memory safety" was a PLT axiom. It's not; it's a software security term of art.

It's been in usage for PLT for at least twenty years[1]. You are at least two decades late to the party.

    Software is memory-safe if (a) it never references a memory location outside the address space allocated by or that entity, and (b) it never executes intstruction outside code area created by the compiler and linker within that address space.

[1]https://llvm.org/pubs/2003-05-05-LCTES03-CodeSafety.pdf

Not GP, but that definition seems not to be the one in use when describing languages like Rust--or even tools like valgrind. Those tools value a definition of "memory safety" that is a superset (a big one) of the definition referenced in that paper: safety as preventing incorrect memory accesses within a program, regardless of whether those accesses are out of bounds/segmentation violations.

...by that definition, can a C program be memory safe as long as it doesn't have any relevant bugs, despite the choice of language? (I realize that in practice, most people are not aware of every bug that exists in their program.)

This is way outside my domain but isn’t the answer: yes, if the code is formally proven safe?

Doesn’t NASA have an incredibly strict, specific set of standards for writing safety critical C that helps with writing programs that can be formalized?

There are safety recommendations / best practice standards like CERT. None of them will prevent you from making intentional looking but logically unsound memory unsafe operations with C and C++. The code can be very indistinguishable from safe code. The things that C and C++ allow you to do basically makes code written in those languages impossible to fully formally prove. Although there are subsets, the basic integer operations and primitive types are messed up with C. So without uprooting how basic integer and pointer types work, it is impossible to make C and C++ safer. Such change will make all C and C++ programs invalid.

C and C++ always defaults to minimum amount of safety for maximum allowance of the compiler interpretation. The priority of the language designers of them is keeping existing terrible code running as long as possible first, letting compilers interpret the source code as freely as possible second.

That's why many military and aerospace code actually uses much safer and significantly more formally verifiable Ada.

I agree that there are two groups here talking past each other. I think it would help a lot to clarify this:

the issue here is that the "Rust and Java sense" of memory safety is not the actual meaning of the term

So what is the actual meaning? Is it simply "there are no cases of actual exploited bugs in the wild"?

Because in another comment you wrote:

a term of art was created to describe something complicated; in this case, "memory safety", to describe the property of programming languages that don't admit to memory corruption vulnerabilities, such as stack and heap overflows, use-after-frees, and type confusions. Later, people uninvolved with the popularization of the term took the term and tried to define it from first principles, arriving at a place different than the term of art.

But type confusion is exactly what has been demonstrated in the post's example. So what kind of memory safety does Go actually provide, in the term of art sense?

It's a contrived type confusion bug. It reads 42h because that address is hardcoded, and it does something that ordinary code doesn't do.

If you were engaged to do a software security assessment for an established firm that used Go (or Python, or any of the other mainstream languages that do shared-memory concurrency and don't have Rust's type system), and you said "this code is memory-unsafe", showing them this example, you would not be taken seriously.

If people want to make PLT arguments about Rust's correctness advantages, I will step out of the way and let them do that. But this article makes a security claim, and that claim is in the practical sense false.

I have recently come to the conclusion that everything I ever thought was "contrived" is currently standard practice in some large presently existing organization.

Take that to the Apple bounty program with your crasher bug and tell them they should pay out as if you'd confirmed RCE, see how it goes. This is an engineering question; it's not about vibes.

It's not even always the case that corrupted data structures (or even pointers) in C code are exploitable. You need attacker control of data and where it goes in memory. It's far less often the case in Python or Go --- in fact, it's basically never the case. As evidence for that claim: the zero memory corruption RCEs in all of shipping Go code, of which there is a lot.

You are however replying to thread where a Dropbox engineer calls it "a right of passage" to introduce such bugs to their codebase. Which suggests that it is by no means unheard of for these problems to crop up in real-world code.

Doesn't Dropbox write a lot of Python extensions in C for speedup?

Excuse me, but in this thread we are bashing go, not making logical arguments.

What is the argument?

Again: introducing surprising correctness bugs? Crashing programs? Absolutely. I don't know how many different ways I can say that my concern here is the misuse of a security term of art. Dropbox engineers do not have as a rite of passage introducing or finding RCE vulnerabilities in Go code. Would that it were so! My job would be much more interesting.

correctness bugs? Crashing programs? Absolutely.

Denial of service can absolutely be a security issue, as can any correctness bug if it leads to unintended behavior or corrupted data.

If that's where we're at, where unhandled exceptions are the security issues we're hanging on, I'll consider my argument won.

It reads 42h because that address is hardcoded,

It is trivial to change this example into an arbitrary int2ptr cast.

Go (or Python, or any of the other mainstream languages that do shared-memory concurrency and don't have Rust's type system),

As the article discusses, only Go has this issue. Python and Java and JavaScript and so on are all memory-safe. Maybe you are mixing up "language has data races" and "data races can cause the language itself to be broken"?

If people want to make PLT arguments about Rust's correctness advantages, I will step out of the way and let them do that. But this article makes a security claim, and that claim is in the practical sense false.

This article makes a claim about the term "memory safety". You are making the claim that that's a security term. I admit I am not familiar with the full history of the term "memory safety", but I do know that "type safety" has been used in PLT for many decades, so it's not like all "safety" terms are somehow in the security domain.

I am curious what your definition of "memory safety" is such that Go satisfies the definition. Wikipedia defines it as

Memory safety is the state of being protected from various software bugs and security vulnerabilities when dealing with memory access, such as buffer overflows and dangling pointers.

My example shows that Go does not enforce memory safety according to that definition -- and not through some sort of oversight or accident, but by design. Out-of-bounds reads and writes are possible in Go. The example might be contrived, but the entire point of memory safety guarantees is that it doesn't matter how contrived the code is.

I'm completely fine with Go making that choice, but I am not fine with Go then claiming to be memory safe in the same sense that Java or Rust are, when it is demonstrably not the case.

The problem isn't that you couldn't hardcode a scarier value; it's that you have to demonstrate a plausible scenario in realistic code where an attacker controls both the value and the address it's written to.

While you're wondering why I keep claiming Go is a memory-safe language, you can also go ask the ISRG, which says the same thing I am at (checks notes) https://www.memorysafety.org/.

While you're wondering why I keep claiming Go is a memory-safe language, you can also go ask the ISRG, which says the same thing I am at

And yet Go violates the definition they give -- it doesn't prevent out-of-bounds accesses. (And just to be sure we're talking about the same thing, I'm specifically talking about Go here. All the other languages on their list are actually memory safe, as far as I know.)

you have to demonstrate a plausible scenario in realistic code where an attacker controls both the value and the address it's written to.

So your definition of memory safety includes some notion of "plausible" and "realistic"? Neither https://www.memorysafety.org/docs/memory-safety/ nor Wikipedia have such a qualification in their definition. It would help if you could just spell out your definition in full, rather than having us guess.

You're just wrong about this. The ability to write contrived code that does an out-of-bounds write, or to induce crashes, doesn't violate the notion of "memory safety" as an ordinary term of art.

So your definition of memory safety includes some notion of "plausible" and "realistic"? Neither https://www.memorysafety.org/docs/memory-safety/ nor Wikipedia have such a qualification in their definition. It would help if you could just spell out your definition in full, rather than having us guess.

This is a strawman argument, you're arguing semantics here. You're a smart person, so you know exactly what he means. The perception created by your article is that people shouldn't use Go because it's not memory-safe. But the average developer hearing "not memory-safe" thinks of C/C++ level issues, with RCEs everywhere.

Unless you can show a realistic way this could be exploited for RCE in actual programs, you're just making noise. Further down the thread, you admit yourself that you're in a PLT research bubble and it shows.

But: everybody understands that.

Everybody does not understand that otherwise there would be zero of these issues in shipping code.

This is the problem with the C++ crowd hoping to save their language. Maybe they'll finally figure out some --disallow-all-ub-and-be-memory-safe-and-thread-safe flag but at the moment it's still insanely trivial to make a mistake and return a reference to some value on the stack or any number of other issues.

The answer can not be "just write flawless code and you'll never have these issues" but at the moment that's all C++, and Go, from this article has.

Again: if you want to make that claim about correctness bugs, that's fine, I get it. But if you're trying to claim that naive Go code has memory safety security bugs: no, that is simply not true.

I cannot find anyone in this thread (nor in the article) making the claim you are arguing against, though... the reason for the example isn't "this demonstrates all Go code is wrong", but merely that "you can't assume that all Go code is correct merely because it is written in Go"; now, most code written in Go might, in fact, be safe, and it might even be difficult to write broken Go code; but, I certainly have come across a LOT of people who are claiming that we don't even have to analyze their code for mistakes because it is written in Go, which is not the case, because people do, in fact, share stuff all over the place and Go, in fact, doesn't prevent you from all possible ways of writing broken code. To convince these people that they have to stop making that assumption requires merely any example of code which fails, and to those people these examples are, in fact, elucidating. Of course, clearly, this isn't showing a security issue, but it isn't claiming to be; and like, obviously, this isn't something that would be sent to a bug bounty program, as who would it even be sent to? I dunno... you seem to have decided you want to win a really minute pedantic point against someone who doesn't exist, and it makes this whole thing very confusing.

The terms correctness (from a PLT perspective) and safety (from a security perspective) are not equivalent and interchangeable. I see them mixed up too much in this discussion.

PLT has used the term "type safety" for a very long time -- so "safety" does not imply a security perspective. And yes it is indeed very different from correctness. But the article doesn't claim that memory safety should imply correctness -- that would be ridiculous, obviously you can write buggy programs in memory-safe languages. The article claims that Go is not memory-safe.

I was referring to comments mentioning correctness and safety as interchangeable terms. The article doesn’t mix them up.

This comment highlights a very important philosophical difference between the Rust community and the communities of other languages:

- in other languages, it’s understood that perhaps the language is vulnerable to certain errors and one should attempt to mitigate them. But more importantly, those errors are one class of bug and bugs can happen. Set up infra to detect and recover.

- in Rust the code must be safe, must be written in a certain way, must be proven correct to the largest extent possible at compile time.

This leads to the very serious, solemn attitude typical of Rust developers. But the reality is that most people just don’t care that much about a particular type of error as opposed to other errors.

... it's understood that perhaps the language is vulnerable to certain errors and one should attempt to mitigate them. But more importantly, those errors are one class of bug and bugs can happen. Set up infra to detect and recover.

in Rust the code must be safe, must be written in a certain way, must be proven correct to the largest extent possible at compile time.

Only for the Safe Rust subset. Rust has the 'unsafe' keyword that shows exactly where the former case does apply. (And even then, only for possible memory unsoundness. Rust does not attempt to fix all possible errors.)

A not-so-secret secret of Rust is that liberal use of 'unsafe' is pretty much required for certain classes of high-performance code.

But: everybody understands that.

I had to convince Go people that you can segfault with Go. Or you mean the language designers with using everybody?

You can segfault in Rust, too - there's a whole subset of the language marked "unsafe" that people ignore when making "safe language" arguments. The question is how difficult is it to have a segfault, and in Go it's honestly pretty hard. It's arguably harder in Rust but it's not impossible.