Expand description
Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree of Rust source code.
Currently this library is geared toward use in Rust procedural macros, but contains some APIs that may be useful more generally.
-
Data structures — Syn provides a complete syntax tree that can represent any valid Rust source code. The syntax tree is rooted at
syn::Filewhich represents a full source file, but there are other entry points that may be useful to procedural macros includingsyn::Item,syn::Exprandsyn::Type. -
Derives — Of particular interest to derive macros is
syn::DeriveInputwhich is any of the three legal input items to a derive macro. An example below shows using this type in a library that can derive implementations of a user-defined trait. -
Parsing — Parsing in Syn is built around parser functions with the signature
fn(ParseStream) -> Result<T>. Every syntax tree node defined by Syn is individually parsable and may be used as a building block for custom syntaxes, or you may dream up your own brand new syntax without involving any of our syntax tree types. -
Location information — Every token parsed by Syn is associated with a
Spanthat tracks line and column information back to the source of that token. These spans allow a procedural macro to display detailed error messages pointing to all the right places in the user’s code. There is an example of this below. -
Feature flags — Functionality is aggressively feature gated so your procedural macros enable only what they need, and do not pay in compile time for all the rest.
§Example of a derive macro
The canonical derive macro using Syn looks like this. We write an ordinary
Rust function tagged with a proc_macro_derive attribute and the name of
the trait we are deriving. Any time that derive appears in the user’s code,
the Rust compiler passes their data structure as tokens into our macro. We
get to execute arbitrary Rust code to figure out what to do with those
tokens, then hand some tokens back to the compiler to compile into the
user’s crate.
[dependencies]
syn = "2.0"
quote = "1.0"
[lib]
proc-macro = true
use proc_macro::TokenStream;
use quote::quote;
use syn::{parse_macro_input, DeriveInput};
#[proc_macro_derive(MyMacro)]
pub fn my_macro(input: TokenStream) -> TokenStream {
// Parse the input tokens into a syntax tree
let input = parse_macro_input!(input as DeriveInput);
// Build the output, possibly using quasi-quotation
let expanded = quote! {
// ...
};
// Hand the output tokens back to the compiler
TokenStream::from(expanded)
}The heapsize example directory shows a complete working implementation
of a derive macro. The example derives a HeapSize trait which computes an
estimate of the amount of heap memory owned by a value.
pub trait HeapSize {
/// Total number of bytes of heap memory owned by `self`.
fn heap_size_of_children(&self) -> usize;
}The derive macro allows users to write #[derive(HeapSize)] on data
structures in their program.
#[derive(HeapSize)]
struct Demo<'a, T: ?Sized> {
a: Box<T>,
b: u8,
c: &'a str,
d: String,
}§Spans and error reporting
The token-based procedural macro API provides great control over where the
compiler’s error messages are displayed in user code. Consider the error the
user sees if one of their field types does not implement HeapSize.
#[derive(HeapSize)]
struct Broken {
ok: String,
bad: std::thread::Thread,
}By tracking span information all the way through the expansion of a
procedural macro as shown in the heapsize example, token-based macros in
Syn are able to trigger errors that directly pinpoint the source of the
problem.
error[E0277]: the trait bound `std::thread::Thread: HeapSize` is not satisfied
--> src/main.rs:7:5
|
7 | bad: std::thread::Thread,
| ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `HeapSize` is not implemented for `Thread`
§Parsing a custom syntax
The lazy-static example directory shows the implementation of a
functionlike!(...) procedural macro in which the input tokens are parsed
using Syn’s parsing API.
The example reimplements the popular lazy_static crate from crates.io as a
procedural macro.
lazy_static! {
static ref USERNAME: Regex = Regex::new("^[a-z0-9_-]{3,16}$").unwrap();
}The implementation shows how to trigger custom warnings and error messages on the macro input.
warning: come on, pick a more creative name
--> src/main.rs:10:16
|
10 | static ref FOO: String = "lazy_static".to_owned();
| ^^^
§Testing
When testing macros, we often care not just that the macro can be used
successfully but also that when the macro is provided with invalid input it
produces maximally helpful error messages. Consider using the trybuild
crate to write tests for errors that are emitted by your macro or errors
detected by the Rust compiler in the expanded code following misuse of the
macro. Such tests help avoid regressions from later refactors that
mistakenly make an error no longer trigger or be less helpful than it used
to be.
§Debugging
When developing a procedural macro it can be helpful to look at what the
generated code looks like. Use cargo rustc -- -Zunstable-options --pretty=expanded or the cargo expand subcommand.
To show the expanded code for some crate that uses your procedural macro,
run cargo expand from that crate. To show the expanded code for one of
your own test cases, run cargo expand --test the_test_case where the last
argument is the name of the test file without the .rs extension.
This write-up by Brandon W Maister discusses debugging in more detail: Debugging Rust’s new Custom Derive system.
§Optional features
Syn puts a lot of functionality behind optional features in order to optimize compile time for the most common use cases. The following features are available.
derive(enabled by default) — Data structures for representing the possible input to a derive macro, including structs and enums and types.full— Data structures for representing the syntax tree of all valid Rust source code, including items and expressions.parsing(enabled by default) — Ability to parse input tokens into a syntax tree node of a chosen type.printing(enabled by default) — Ability to print a syntax tree node as tokens of Rust source code.visit— Trait for traversing a syntax tree.visit-mut— Trait for traversing and mutating in place a syntax tree.fold— Trait for transforming an owned syntax tree.clone-impls(enabled by default) — Clone impls for all syntax tree types.extra-traits— Debug, Eq, PartialEq, Hash impls for all syntax tree types.proc-macro(enabled by default) — Runtime dependency on the dynamic library libproc_macro from rustc toolchain.
Modules§
- A stably addressed token buffer supporting efficient traversal based on a cheaply copyable cursor.
- Extension traits to provide parsing methods on foreign types.
- Facility for interpreting structured content inside of an
Attribute. - Parsing interface for parsing a token stream into a syntax tree node.
- A punctuated sequence of syntax tree nodes separated by punctuation.
- A trait that can provide the
Spanof the complete contents of a syntax tree node. - Tokens representing Rust punctuation, keywords, and delimiters.
- Syntax tree traversal to mutate an exclusive borrow of a syntax tree in place.
Macros§
- A type-macro that expands to the name of the Rust type representation of a given token.
- Parse a set of curly braces and expose their content to subsequent parsers.
- Parse a set of square brackets and expose their content to subsequent parsers.
- Define a type that supports parsing and printing a given identifier as if it were a keyword.
- Define a type that supports parsing and printing a multi-character symbol as if it were a punctuation token.
- Parse a set of parentheses and expose their content to subsequent parsers.
- Parse the input TokenStream of a macro, triggering a compile error if the tokens fail to parse.
- Quasi-quotation macro that accepts input like the
quote!macro but uses type inference to figure out a return type for those tokens. - This macro is
parse_quote!+quote_spanned!.
Structs§
- The binary interface of a function:
extern "C". - Angle bracketed arguments of a path segment: the
<K, V>inHashMap<K, V>. - One arm of a
matchexpression:0..=10 => { return true; }. - An equality constraint on an associated constant: the
PANIC = falseinTrait<PANIC = false>. - A binding (equality constraint) on an associated type: the
Item = u8inIterator<Item = u8>. - An attribute, like
#[repr(transparent)]. - An argument in a function type: the
usizeinfn(usize) -> bool. - The variadic argument of a function pointer like
fn(usize, ...). - A braced block containing Rust statements.
- A set of bound lifetimes:
for<'a, 'b, 'c>. - A const generic parameter:
const LENGTH: usize. - An associated type bound:
Iterator<Item: Display>. - An enum input to a
proc_macro_derivemacro. - A struct input to a
proc_macro_derivemacro. - An untagged union input to a
proc_macro_derivemacro. - Data structure sent to a
proc_macro_derivemacro. - Error returned when a Syn parser cannot parse the input tokens.
- A slice literal expression:
[a, b, c, d]. - An assignment expression:
a = compute(). - An async block:
async { ... }. - An await expression:
fut.await. - A binary operation:
a + b,a += b. - A blocked scope:
{ ... }. - A
break, with an optional label to break and an optional expression. - A function call expression:
invoke(a, b). - A cast expression:
foo as f64. - A closure expression:
|a, b| a + b. - A const block:
const { ... }. - A
continue, with an optional label. - Access of a named struct field (
obj.k) or unnamed tuple struct field (obj.0). - A for loop:
for pat in expr { ... }. - An expression contained within invisible delimiters.
- An
ifexpression with an optionalelseblock:if expr { ... } else { ... }. - A square bracketed indexing expression:
vector[2]. - The inferred value of a const generic argument, denoted
_. - A
letguard:let Some(x) = opt. - A literal in place of an expression:
1,"foo". - Conditionless loop:
loop { ... }. - A macro invocation expression:
format!("{}", q). - A
matchexpression:match n { Some(n) => {}, None => {} }. - A method call expression:
x.foo::<T>(a, b). - A parenthesized expression:
(a + b). - A path like
std::mem::replacepossibly containing generic parameters and a qualified self-type. - A range expression:
1..2,1..,..2,1..=2,..=2. - Address-of operation:
&raw const placeor&raw mut place. - A referencing operation:
&aor&mut a. - An array literal constructed from one repeated element:
[0u8; N]. - A
return, with an optional value to be returned. - A struct literal expression:
Point { x: 1, y: 1 }. - A try-expression:
expr?. - A try block:
try { ... }. - A tuple expression:
(a, b, c, d). - A unary operation:
!x,*x. - An unsafe block:
unsafe { ... }. - A while loop:
while expr { ... }. - A yield expression:
yield expr. - A field of a struct or enum variant.
- A single field in a struct pattern.
- A field-value pair in a struct literal.
- Named fields of a struct or struct variant such as
Point { x: f64, y: f64 }. - Unnamed fields of a tuple struct or tuple variant such as
Some(T). - A complete file of Rust source code.
- A foreign function in an
externblock. - A macro invocation within an extern block.
- A foreign static item in an
externblock:static ext: u8. - A foreign type in an
externblock:type void. - Lifetimes and type parameters attached to a declaration of a function, enum, trait, etc.
- A word of Rust code, which may be a keyword or legal variable name.
- Returned by
Generics::split_for_impl. - An associated constant within an impl block.
- An associated function within an impl block.
- A macro invocation within an impl block.
- An associated type within an impl block.
- The index of an unnamed tuple struct field.
- A constant item:
const MAX: u16 = 65535. - An enum definition:
enum Foo<A, B> { A(A), B(B) }. - An
extern crateitem:extern crate serde. - A free-standing function:
fn process(n: usize) -> Result<()> { ... }. - A block of foreign items:
extern "C" { ... }. - An impl block providing trait or associated items:
impl<A> Trait for Data<A> { ... }. - A macro invocation, which includes
macro_rules!definitions. - A module or module declaration:
mod mormod m { ... }. - A static item:
static BIKE: Shed = Shed(42). - A struct definition:
struct Foo<A> { x: A }. - A trait definition:
pub trait Iterator { ... }. - A trait alias:
pub trait SharableIterator = Iterator + Sync. - A type alias:
type Result<T> = std::result::Result<T, MyError>. - A union definition:
union Foo<A, B> { x: A, y: B }. - A use declaration:
use std::collections::HashMap. - A lifetime labeling a
for,while, orloop. - A Rust lifetime:
'a. - A lifetime definition:
'a: 'b + 'c + 'd. - A boolean literal:
trueorfalse. - A byte literal:
b'f'. - A byte string literal:
b"foo". - A nul-terminated C-string literal:
c"foo". - A character literal:
'a'. - A floating point literal:
1f64or1.0e10f64. - An integer literal:
1or1u16. - A UTF-8 string literal:
"foo". - A local
letbinding:let x: u64 = s.parse()?. - The expression assigned in a local
letbinding, including optional divergingelseblock. - A macro invocation:
println!("{}", mac). - A structured list within an attribute, like
derive(Copy, Clone). - A name-value pair within an attribute, like
feature = "nightly". - Arguments of a function path segment: the
(A, B) -> CinFn(A,B) -> C. - A const block:
const { ... }. - A pattern that binds a new variable:
ref mut binding @ SUBPATTERN. - A literal in place of an expression:
1,"foo". - A macro invocation expression:
format!("{}", q). - A pattern that matches any one of a set of cases.
- A parenthesized pattern:
(A | B). - A path like
std::mem::replacepossibly containing generic parameters and a qualified self-type. - A range expression:
1..2,1..,..2,1..=2,..=2. - A reference pattern:
&mut var. - The dots in a tuple or slice pattern:
[0, 1, ..]. - A dynamically sized slice pattern:
[a, b, ref i @ .., y, z]. - A struct or struct variant pattern:
Variant { x, y, .. }. - A tuple pattern:
(a, b). - A tuple struct or tuple variant pattern:
Variant(x, y, .., z). - A type ascription pattern:
foo: f64. - A pattern that matches any value:
_. - A path at which a named item is exported (e.g.
std::collections::HashMap). - A segment of a path together with any path arguments on that segment.
- Precise capturing bound: the ‘use<…>’ in
impl Trait + use<'a, T>. - A lifetime predicate in a
whereclause:'a: 'b + 'c. - A type predicate in a
whereclause:for<'c> Foo<'c>: Trait<'c>. - The explicit Self type in a qualified path: the
Tin<T as Display>::fmt. - The
selfargument of an associated method. - A function signature in a trait or implementation:
unsafe fn initialize(&self). - A macro invocation in statement position.
- A trait used as a bound on a type parameter.
- An associated constant within the definition of a trait.
- An associated function within the definition of a trait.
- A macro invocation within the definition of a trait.
- An associated type within the definition of a trait.
- Returned by
TypeGenerics::as_turbofish. - A fixed size array type:
[T; n]. - A bare function type:
fn(usize) -> bool. - Returned by
Generics::split_for_impl. - A type contained within invisible delimiters.
- An
impl Bound1 + Bound2 + Bound3type whereBoundis a trait or a lifetime. - Indication that a type should be inferred by the compiler:
_. - A macro in the type position.
- The never type:
!. - A generic type parameter:
T: Into<String>. - A parenthesized type equivalent to the inner type.
- A path like
std::slice::Iter, optionally qualified with a self-type as in<Vec<T> as SomeTrait>::Associated. - A raw pointer type:
*const Tor*mut T. - A reference type:
&'a Tor&'a mut T. - A dynamically sized slice type:
[T]. - A trait object type
dyn Bound1 + Bound2 + Bound3whereBoundis a trait or a lifetime. - A tuple type:
(A, B, C, String). - A glob import in a
useitem:*. - A braced group of imports in a
useitem:{A, B, C}. - An identifier imported by a
useitem:HashMap. - A path prefix of imports in a
useitem:std::.... - An renamed identifier imported by a
useitem:HashMap as Map. - The variadic argument of a foreign function.
- An enum variant.
- A visibility level restricted to some path:
pub(self)orpub(super)orpub(crate)orpub(in some::module). - A
whereclause in a definition:where T: Deserialize<'de>, D: 'static.
Enums§
- Distinguishes between attributes that decorate an item and attributes that are contained within an item.
- A binary operator:
+,+=,&. - Single parameter in a precise capturing bound.
- The storage of a struct, enum or union data structure.
- A Rust expression.
- Unused, but reserved for RFC 3323 restrictions.
- Data stored within an enum variant or struct.
- An argument in a function signature: the
n: usizeinfn f(n: usize). - An item within an
externblock. - An individual generic argument, like
'a,T, orItem = T. - A generic type parameter, lifetime, or const generic:
T: Into<String>,'a: 'b,const LEN: usize. - An item within an impl block.
- Unused, but reserved for RFC 3323 restrictions.
- Things that can appear directly inside of a module or scope.
- A Rust literal such as a string or integer or boolean.
- A grouping token that surrounds a macro body:
m!(...)orm!{...}orm![...]. - A struct or tuple struct field accessed in a struct literal or field expression.
- Content of a compile-time structured attribute.
- A pattern in a local binding, function signature, match expression, or various other places.
- Angle bracketed or parenthesized arguments of a path segment.
- Mutability of a raw pointer (
*const T,*mut T), in which non-mutable isn’t the implicit default. - Limit types of a range, inclusive or exclusive.
- Return type of a function signature.
- The mutability of an
Item::StaticorForeignItem::Static. - A statement, usually ending in a semicolon.
- A modifier on a trait bound, currently only used for the
?in?Sized. - An item declaration within the definition of a trait.
- The possible types that a Rust value could have.
- A trait or lifetime used as a bound on a type parameter.
- A unary operator:
*,!,-. - A suffix of an import tree in a
useitem:Type as Renamedor*. - The visibility level of an item: inherited or
puborpub(restricted). - A single predicate in a
whereclause:T: Deserialize<'de>.
Functions§
- Parse tokens of source code into the chosen syntax tree node.
- Parse a proc-macro2 token stream into the chosen syntax tree node.
- Parse the content of a file of Rust code.
- Parse a string of Rust code into the chosen syntax tree node.
Type Aliases§
- The result of a Syn parser.