Implementing the Bitfields Proc-Macro [RUST]

Keyboard shortcuts Press ← or → to navigate between chapters Press S or / to search in the book Press ? to show this help Press Esc to hide this help Writing the Bitflags Macro As you may recall from the previous chapter, we used a proc-macro that was called bitfields. In this chapter, we are going to learn about Rust’s procedural macros and even implement one ourselves. Another great resource for this subject is the great video Comprehending Proc Macros by Logan Smith If you are familiar with procedural macros, syn and quote, and want to go straight to the macro implementation, click here A Little Introduction to Procedural Macros Macros are not a new idea in programming languages, and most languages have them in some form. But what even is a macro? If you ask Wikipedia, we get the following definition. Macro A macro is a rule or pattern that specifies how a certain input should be mapped to a replacement output. When I read this definition, the first thing that comes to mind is that it really sounds like a function. After all, a function maps the input arguments to the output arguments, which is exactly what a macro does. And that is exactly right. Rust, macros (specifically procedural macros), are indeed a specific type of function, but let’s not get ahead of ourselves. The key differences between macros and regular functions are that macros replace the inputs and the outputs, and that is not always true with functions. Secondly, macros operate on our source code instead of variables in our program. Rust takes this definition very literally, and the definition for a proc-macro function looks like this: #[proc_macro] pub fn custom_proc_macro(input: TokenStream) -> TokenStream { eprintln!("{:?}", input); input } As you can see in this function, the input is Rust’s TokenStream, which is literally our source code, and the output is also a TokenStream, which means it expects us to return also source code, which could be the same (Like the example above), but most of the time it is not. But what is this TokenStream? Why not just use strings of the source code? Well, the main reason we are even discussing this is that we want to manipulate the initial code in some way. Tokenizing the source code allows us to manipulate the code at a higher level, which is easier to reason about.

This TokenStream is the most basic tokenization unit that we are going to work with, and it contains a sequence of TokenTree nodes that represent the source code. /// A single token or a delimited sequence of token trees (e.g., `[1, (), ..]`). #[stable(feature = "proc_macro_lib2", since = "1.29.0")] #[derive(Clone)] pub enum TokenTree { /// A token stream surrounded by bracket delimiters. #[stable(feature = "proc_macro_lib2", since = "1.29.0")] Group(#[stable(feature = "proc_macro_lib2", since = "1.29.0")] Group), /// An identifier. #[stable(feature = "proc_macro_lib2", since = "1.29.0")] Ident(#[stable(feature = "proc_macro_lib2", since = "1.29.0")] Ident), /// A single punctuation character (`+`, `,`, `$`, etc.). #[stable(feature = "proc_macro_lib2", since = "1.29.0")] Punct(#[stable(feature = "proc_macro_lib2", since = "1.29.0")] Punct), /// A literal character (`'a'`), string (`"hello"`), number (`2.3`), etc. #[stable(feature = "proc_macro_lib2", since = "1.29.0")] Literal(#[stable(feature = "proc_macro_lib2", since = "1.29.0")] Literal), } To see this more visibly, we can print our TokenStream, because it implements the Debug trait. Which, for a simple struct, would look like this: TokenStream [ Ident { ident: "struct", span: #0 bytes(43..49), }, Ident { ident: "Example", span: #0 bytes(50..57), }, Group { delimiter: Brace, stream: TokenStream [ Ident { ident: "a", span: #0 bytes(64..65),

}, Punct { ch: ':', spacing: Alone, span: #0 bytes(65..66), }, Ident { ident: "i32", span: #0 bytes(67..70), }, Punct { ch: ',', spacing: Alone, span: #0 bytes(70..71), }, ], span: #0 bytes(58..73), }, ] Can you understand the name of the struct and its fields? How Macros are Executed As you may have noticed, macros do not behave exactly like regular functions. Another difference that they have is that they are evaluated at compile time. This thinking can also be used on regular functions, but not from our point of view, but from the compiler’s point of view. For the compiler, regular functions are also a mapping, from some target language (in our case, Rust) to some other target language (in most cases, ASM1). For example, this function: #[unsafe(no_mangle)] pub fn square(num: i32) -> i32 { num * num } would map to the following ASM code: Tip Look it yourself at compiler explorer square: mov eax, edi imul eax, edi ret From this point of view, macros are not so different, but instead of a target language, they are mapped to the same language. So this macro: macro_rules! square { ($num:expr) => { $num * $num }; } fn foo() -> u32 { let x: u32 = 42; square!(x) } Would map to this literal Rust code: fn foo_expanded() -> u32 { let x: u32 = 42; x * x } The fact that macros operate on our source code means that we can abstract certain logics that regular functions cannot. For example, take a look at this macro: macro_rules! unwrap_or_break { ($e:expr) => { match $e { Some(v) => v, None => break, } }; } fn main() { let data: Vec<Option<i32>> = vec![Some(1), Some(2), None, Some(4)]; for d: Option<i32> in data { let val: i32 = unwrap_or_break!(d); // breaks the loop on None println!("{}",

val); } println!("done"); } It works because it injects the break expression into the code at the call site, which is something that a function just can’t do. fn unwrap_or_break<T>(e: Option<T>) -> T { match e { Some(v: T) => v, None => break, // ERROR: `break` outside of a loop } } At this time, I hope you understand the great power of macros, and the great code generation capabilities that they enable. But, you might think rightfully think that in the examples above, we didn’t have the option to insert ‘coding’ logic into the macro expansion. This is where procedural macros come in. Macro Types Just before we dive into procedural macros, let’s cover the type of macro that we already used in the examples above. All the syntax information about how macros are structured is taken directly from the Official Rust Reference. Declarative Macros Declarative macros are the simplest type of macro, and they are the ones that we used in the examples above. They are mainly used to generate simple syntax extensions, which are commonly called “macros by example”. Each macro is defined by a set of rules that specify how the macro should expand. Each rule looks a bit like a function signature that can get certain Metavariables. These Metavariables are placeholders for certain Rust syntax that are replaced with actual values when the macro is expanded. Let’s analyze the syntax of a declarative macro rule from the earlier examples. /// Macros are defined using the `macro_rules!` macro, /// followed by the name of the macro. macro_rules! unwrap_or_break { // Each rule is defined with the "() => {}" syntax, // in the parentheses we provide the pattern to match, // which uses `Metavariables` to capture parts of the input. ($e:expr) => { // Then, we can write 'regular' Rust code inside the macro body, // which uses the metavariables to generate the expanded code. match $e { Some(v) => v, None => break, } }; } We will go a bit deeper than necessary on the common types of metavariables that are available. This is because later in this chapter, we are going to talk about the syn library, which will parse Rust’s syntax into similar structures.

Each metavariable starts with a $ followed by the name of the metavariable, which is used to refer to it. Then it is followed by a colon and the type of the metavariable. The common types of metavariables are: Idents ($i:ident) => These can be function names, variable names, type names, etc. They also include keywords like fn, let, struct, etc. Expressions ($e:expr) => Expressions are things that are evaluated to a value, like 1 + 2 or foo.bar(). Items ($i:item) => Items are the components of a module, for example the entire definition of a function or a struct. Statements ($s:stmt) => Statements are the individual lines of code that make up a function or block. For example, let x = 42; is a statement. Blocks ($b:block) => Blocks are groups of statements that are executed on the same scope. For example, { let y = 33; let x = 7 + y; x } is a block. For a full list of available metavariable types, see the reference Procedural Macros Now for the real deal. Procedural macros give us the ability to go beyond simple syntax extensions and allow us to write custom Rust code that will run at compile time on the macro input to consume and produce new Rust syntax (Depending on the macro type, the returned syntax will replace the input syntax or will be added to it). Because procedural macros are another piece of code that will run at compile time, they cannot be defined in the same crate as the code that uses them. This is because the Rust compiler must initially compile the code of the macro so it will be able to run it during the compilation process. In addition, each proc macro crate must add the following configuration to its Cargo.toml file, which will tell Cargo that this is a proc macro crate. [lib] proc-macro = true Like all functions, these macro functions can also fail, although these functions are allowed to panic. They are encouraged to use the compile_error! macro to return a compile-time error instead, which is the compiler form of panic! To gain the Tokenstream type and the attributes that will be used on the macro functions, we will use the proc_macro crate, which is automatically linked to our crate if it is a proc macro crate.

function_like!() Function like macros are very similar to declarative macros. They are invoked like a regular function and take a TokenStream as input and return a TokenStream as output. This type of macro can be called anywhere in our code, even in the global scope, and is defined using the following syntax: #[proc_macro] pub fn foo(_item: TokenStream) -> TokenStream { "fn bar() -> u32 { 42 }".parse().unwrap() } Then it can be called like a regular function, which will create a function that is called bar which could be used in our code. use crate::foo; foo!(); fn main() { println!("{}", bar()); } This type of macro replaces the macro invocation with the generated code, so the macro invocation is effectively replaced with the generated code. #[derive(CustomDerive)] Derive macros are used on Rust items to generate code automatically. They are invoked using the #[derive] attribute and take the item they are applied to as input. Most of the time, derive macros are used to implement traits such as Debug, Clone, PartialEq, etc. Derives may also include helper attributes, which are used to customize the generated code. This type of macro can be called only from structs, enums, or unions. #[proc_macro_derive(WithHelperAttr, attributes(helper))] pub fn derive_with_helper_attr(_item: TokenStream) -> TokenStream { TokenStream::new() } And it is used on a structure like this: #[derive(WithHelperAttr)] struct Foo { #[helper] bar: (), } This type of macro does not replace the macro invocation or the input item with the generated code, and the generated TokenStream is appended to the input TokenStream. #[attribute(macros)] Attributes are used to annotate items. They are placed before the item they are applied to and are used to customize the behavior of the item. Attributes may also include input variables, which can be used to pass ‘configuration’ to the macro. #[proc_macro_attribute] pub fn return_as_is(_attr: TokenStream, item: TokenStream) -> TokenStream { item } And it is used on a structure like this: #[return_as_is] struct Bar { foo: (), } #[return_as_is] fn bar() {} This type of macro replaces the macro invocation and the input item with the generated code.