Kanye West feared Kim Kardashian was going to leave him after controversial slavery comments