Warning: The following definition of legalism may upset some people. Legalism is not just "going through the motions" of the Christian life, nor is it only a problem in the church down the street which has more rules than we do.
Legalism, is the belief and attempt to become more spiritual by following a prescribed set of principles and standards. If that sounds like the way most of us were taught to live as a Christian, you would be right. Legalism is pervasive, and most of us have been well-trained in approaching the Christian life in a legalistic manner. We extract principle from Scripture and do our best to apply them to our attitudes and behavior. That is an attempt to find life by following external "laws" and it differs very little from what most people did under the Old Covenant.
Immediately, many people object to this definition on two counts. First, they want to know if that means we should just do whatever we want and ignore God's laws. Second, they wonder how else one might go about becoming a better person, if not by following the principles laid down in Scripture.
The first objection is simply a false dichotomy. It assumes that there are only two options in life: (1) Do what is right because it is right; or (2) Ignore what is right and do something else. Everything boils down to obedience and disobedience. Unfortunately, this is far too simplistic for real life (it is even non-Scriptural), and it would take a book to explain all the distortions introduced into the Christian life by this approach. But a short response is that there is a third alternative — Be transformed on the inside so that you live out God's way of life by virtue of who you are, not by forcing yourself to follow a set of laws. How that might happen is a topic for another time, and it falls under the heading of "Spiritual Formation" (How we can be formed into the people God created us to be). The second objection then is covered by our response to the first. Once we know how to engage with God to be changed from the inside out, God's laws are then "written on our heart" and we live out of the person we are becoming rather than by trying to adhere to a set of principles.
But now to the upside of legalism. Surely doing the right thing even when we do not feel like it is a good thing, and abstaining from doing something bad even when we have some desire to do so is also a good thing. On this we can all agree. I am at least not violating God's ways. I am not adding more sin to my life and further damaging my world. That is good. And in the absence of inner transformation, it is far better to do what is right than to follow our own desires. To live by desire alone is disastrous.
The question is not whether such action is good. The question is whether or not it is adequate and whether it makes me more "spiritual." When Jesus said that our righteousness must exceed that of the Pharisees, He was drawing a distinction between the virtue of an action and the condition of the person. For Jesus, righteousness is about the condition of a person's soul, not just their outward behavior. If I have to force myself to do the right thing or willfully restrain myself from doing something wrong, then by definition I have a problem in my soul. While it is good to not give into our baser instincts, God wants to go beyond compliance and actually free us from the tyranny of our own inner wounds and desires. He wants to literally alter our inner life in such a way that we truly prefer the good without reservation and truly despise the bad without ambivalence.
So you see the downside of legalism is that we might mistake our compliance for the real thing. We might think that by doing the right things we have done what God wants, and that is all there is to it. Many people have even given up on inner transformation because they have tried to change their heart without much success. But that only means they need more help engaging with God for the kind of change God wants to do in them. And if we settle for following rules and principles, we place ourselves in serious danger. Not only can we start feeling self-righteous, but we can actually begin to mal-form our soul in destructive ways.
If over time I begin to resent the good that I do, or if I feel cheated by abstaining from things I really wish I could indulge in, my heart may become convinced that we serve a God of deprivation rather than a God of abundance. And if I continue to run roughshod over my feelings in order to do the right things, I may eventually lose touch with those feelings and begin to believe they have been tamed, when in fact they have merely gone underground. I know a man quite well who has practiced self-righteousness all his life, with tremendous contempt and disgust for anyone who struggles or fails. But when I confronted him once about his obvious contempt for everyone and everything around him, he flatly denied that he had any contempt at all. He could no longer feel it or identify it in himself. And therein lies the fruit of legalism.
So while we should always strive to do all that God has commanded, we must at the same time take notice of when we secretly wish we could do otherwise, or when we have a disagreement between our head and our heart, or when we are doing what we do without any real joy. Those are all signals that God wants to do much more in us that we cannot do for ourselves. We need to seek God for more renewal. And in that way, His law can lead us back to Him.