Borrow checker treating logically immutable reference as mutable

Is there a good reason for the following code to not compile, or is it just a limitation of the borrow checker? And if it is a limit of the borrow checker, is it feasible to un-limit it at some point or should i rely on unsafe blocks? fn main() { let mut x: i32 = 5; let x_ref: &i32; { let x_ref_mut = &mut x; x_ref = &*x_ref_mut; } &x; x_ref; } error[E0502]: cannot borrow `x` as immutable because it is also borrowed as mutable --> src/main.rs:8:5 | 5 ...