First, your question about drivability, will it affect your XJ's drivability?...... Of course it will. The sway bar was installed on your vehicle to help it's road manners and help Chrysler stay out of lawsuits when their SUVs end up riding on their lids (does anyone remember 60 Minutes and the CJs scare?). The vehicle, any vehicle, has better cornering and body roll characteristics with the employ of some sort of suspension eqalization product.
Reality is this, yes your Jeep will drive fine, no parts will break as a direct result of the sway bar being disconnected, and your offroad prowess will greatly, and I mean greatly, improve. Just remember what the bar was there for, and don't drive the Jeep in such a way as to exploit the now weaker street-ability. If you have to swerve for a cat on the freeway... well... you better hit the cat.
If this is your driver, do not disconnect the front bar. Spend the money on a decent disconnect system. Mepco carries an awesome one. That way, like the others have mentioned above, you can connect it to drive to the canyon, then disconnect it while wheelin', then connect and shimmy on home.
An interesting note is that you are seeing more an more of the long travel sway bars appear off road, on buggies and other rigs... these are usually quite supple and don't hinder the suspension too much, but they really control the back on and forth rocking that you are going to soon be familiar with
Mepco sells the Tera one that can also be disconnected if desired, and Currie Enterprises has the Anti-Rock... As for me, I really, really want to go grab the Tera version.
-JJ