A CharacterController will not be able to be pushed around by physics (which can be positive or negative). It may move slightly smoother across the ground (as it has been specifically designed to do that well). It is also easier for a rigidbody to realistically push other physics objects out of the way.
In general the CharacterController is slightly slower than a rigidbody + colldier, in particular when many of them are colliding.
When using a rigidbody the movement code will run during FixedUpdate instead of during Update.