big wheels, big brakes?

If I put 24's on my ride, is it a must to upgrade my brake system? I heard you can f*ck up your brakes being that the wheels are so big and heavy.