I am assuming you have already removed the floating sheave/spring etc..
A penetrent applied through the set screw hole and the outer end of the bore.
Soak over nite.
likely be necessary.
Just do so only close to the shaft and all the way around it
A hard wood wedge may
be useful.(see last
If stubborn, something harder may be required.
Try lighter forces first and increase gradually until it budges.
Now how do you propose to correct the rubbing?