95% of My PySpark Job Finished in 4 Minutes. The Last Task Took 40. Here's Why.

I had a PySpark job joining a large transactions table with a customer dimension table. Nothing...