The Vale Programming Language

Memory safety is an incredibly useful aspect for a programming language. It protects us from maddening bugs, devious vulnerabilities, and cantankerous shenanigans.

Getting memory safety with predictable performance 0 is quite a challenge! Most languages need to sacrifice memory safety for other features, such as FFI or unsafe blocks.

We've discovered a better way to offer memory safety.

Generational References' Hidden Superpower

Vale starts with a foundation of generational references. A generational reference is when every object has a "current generation" integer which we increment on free, and every pointer has a "remembered generation" number from the object. When we dereference a pointer, we assert that those two numbers still match. See Generational References for a more in-depth explanation!

Most other languages use garbage collection, reference counting, or borrow checking, but we chose generational references because they're fast 1 and more flexible than borrow checking, allowing us to use common safe patterns like observers, higher RAII, the dependency injection pattern, delegates, back-references, graphs, and so on.

We also discovered that they enable Vale to have complete memory safety, something no native language has been able to achieve.

This is because generational references:

  • Are not reference counted, so if we give one to some extern C code, we don't need to trust it to maintain reference counts to uphold our memory safety. 2
  • Are able to detect when an object has been deallocated.
  • Require no unsafe blocks, because there is no borrow checker we need to work around.

Generational references are pretty stellar for memory safety. However, there's one remaining problem: what about FFI?

FFI and Region Isolation

Foreign Function Interface (FFI) is when a language allows calling into another language's code. For example, Objective-C might make a pointer to some data and pass it to C, which might have a bug which corrupts that data.

This is called "leaky safety", and its bugs are very difficult to track down, because their symptoms manifest so far from their cause.

This can also happen when a language has unsafe escape hatches. If some unsafe code corrupts some memory, it can cause undefined behavior in safe code. For example, see this Rust snippet where an unsafe block corrupts some memory that's later used by the safe code.

In all these cases, we know that the unsafe language was involved somewhere in the chain of events, but since the bugs actually happen later on, in supposedly safe code, there's no easy way to identify which unsafe code was the original culprit.

To solve this, Vale has Fearless FFI which decouples and isolates unsafe C data from safe Vale data.

  • Separate the safe memory from the unsafe memory (such as the memory managed by C). This includes:
    • Not allowing safe objects to contain unsafe objects.
    • Not allowing unsafe objects to contain safe objects.
    • Using a different stack for the unsafe code.
  • Allowing references between the two:
    • A safe object can contain a reference to an unsafe object.
    • An unsafe object can contain a reference to a safe object, and it's automatically scrambled.
  • Enable passing memory between the two by copying, also known as message passing.

See Fearless FFI for more on this!

This protects us from any bugs in C that might otherwise accidentally corrupt our Vale data. 3

Performance

Generational references are very fast, but if we want that extra sliver of performance, Vale plans to add the Check Override Operator.

Note that the check override operator, region borrow checker, and hybrid-generational memory are all upcoming features, not available yet. We mention them here to show our plans for balancing safety with speed!

Most generation checks will be skipped by the region borrow checker and Hybrid-Generational Memory. For example, we recently implemented a Cellular Automata algorithm, and by our measurements, regions and hybrid-generational memory would eliminate every single generation check in the entire algorithm.

Still, for the occasional generation check that those two might not eliminate, we have the Check Override Operator, which will skip the generation check for a generational reference.

"But what if a dependency uses a check override operator, and causes memory unsafety? Isn't this just as bad as an unsafe block?"

It would seem so, except for one key detail: the check override operator is ignored by default for any dependencies. One must explicitly enable a dependency to ignore its checks.

Most people will be using it with checks on, so everyone will find out very quickly if there's any unsafe behavior in practice. Anyone who wants that extra sliver of performance can then opt-in to skipping the checks in release mode, with a little more confidence that unsafety will be detected by other users of the library, or in development or testing.

And if a user isn't comfortable with that for their situation, they simply stick with the defaults, which ignore the check override operator.

We think this is a perfect tradeoff to allow memory safety when it's critical, while not compromising the safety of the language or ecosystem.

We're also thinking of adding a way to skip all generation checks for a given block of code. 4

Tying it All Together

We talked about three mechanisms:

  • Generational references
  • Fearless FFI
  • Check Override Operator

With these measures in place, Vale will be the first completely memory safe native language!

If you're impressed with our track record so far and believe in the direction we're heading, please consider sponsoring us:

With your help, we can bring complete memory safety to programmers everywhere!

Side Notes
(interesting tangential thoughts)
0

Tracing garbage collection (like in Java) is a great solution for memory safety, but there are cases where more predictable performance is highly desirable, such as in embedded devices, games, or certain kinds of servers.

1

They're not just fast, but they could get even faster when we introduce regions and Hybrid-Generational Memory

2

For example, when Python code sends a Python object into C, if the C code doesn't correctly call Py_INCREF, it will corrupt Python's memory and cause some mysterious behavior later on in the Python code.

3

We could also protect against malicious code with sandboxing, via webassembly or subprocesses. This is a planned feature, see Fearless FFI for more on this!

4

We might even call it an unsafe block, if that doesn't cause confusion with other languages.

Vale's Vision

Vale aims to bring a new way of programming into the world that offers speed, safety, and ease of use.

The world needs something like this! Currently, most programming language work is in:

  • High-overhead languages involving reference counting and tracing garbage collection.
  • Complex languages (Ada/Spark, Coq, Rust, Haskell, etc.) which impose higher complexity burden and mental overhead on the programmer.

These are useful, but there is a vast field of possibilities in between, waiting to be explored!

Our aim is to explore that space, discover what it has to offer, and make speed and safety easier than ever before.

In this quest, we've discovered and implemented a lot of new techniques:

  • Generational Memory, for a language to ensure an object still exists at the time of dereferencing.
  • Higher RAII, a form of linear typing that enables destructors with parameters and returns.
  • Fearless FFI, which allows us to call into C without risk of accidentally corrupting Vale objects.
  • Perfect Replayability, to record all inputs and replay execution, and completely solve heisenbugs and race bugs.

These techniques have also opened up some new emergent possibilities:

  • Region Borrow Checking, which adds mutable aliasing support to a Rust-like borrow checker.
  • Hybrid-Generational Memory, which ensures that nobody destroys an object too early, for better optimizations.
  • Seamless concurrency, the ability to launch multiple threads that can access any pre-existing data without data races, without the need for refactoring the code or the data.
  • Object pools and bump-allocators that are memory-safe and decoupled, so no refactoring needed.

We also gain a lot of inspiration from other languages, and are finding new ways to combine their techniques:

  • We can mix an unsafe block with Fearless FFI to make a much safer systems programming language!
  • We can mix Erlang's isolation benefits with functional reactive programming to make much more resilient programs!
  • We can mix region borrow checking with Pony's iso to support shared mutability.

...plus a lot more interesting ideas to explore!

The Vale programming language is a novel combination of ideas from the research world and original innovations. Our goal is to publish our techniques, even the ones that couldn't fit in Vale, so that the world as a whole can benefit from our work here, not just those who use Vale.

Our medium-term goals:

  • Finish the Region Borrow Checker, to show the world that shared mutability can work with borrow checking!
  • Prototype Hybrid-Generational Memory in Vale, to see how fast and easy we can make single ownership.
  • Publish the Language Simplicity Manifesto, a collection of principles to keep programming languages' learning curves down.
  • Publish the Memory Safety Grimoire, a collection of "memory safety building blocks" that languages can potentially use to make new memory models, just like Vale combined generational references and scope tethering.

We aim to publish articles biweekly on all of these topics, and create and inspire the next generation of fast, safe, and easy programming languages.

If you want to support our work, please consider sponsoring us on GitHub!

With enough sponsorship, we can:

  • Work on this full-time.
  • Turn the Vale Language Project into a 501(c)(3) non-profit organization.
  • Make Vale into a production-ready language, and push it into the mainstream!