I just finished another of his books - Orthodoxy - which is one of his earliest works. It's so old (1908), in fact, that the binding of the one I borrowed is disintegrating....and that the copyright seems to have expired. The book is online for free!
Chesterton's writing doesn't make for good excerpts; nonetheless here are the only passages that I transcribed into my notes. In no way do they summarize or encapsulate this book. In fact, Wikipedia and other places on the web will do a much better job.
Poetry is sane because it floats easily in an infinite sea; reason seeks to cross the infinite sea, and so make it finite. The result is mental exhaustion...To accept everything is an exercise, to understand everything a strain. The poet only desires exaltation and expansion, a world to stretch himself in. The poet only asks to get his head into the heavens. It is the logician who seeks to get the heavens into his head. And it is his head that splits. (p29)
The whole secret of mysticism is this: that man can understand everything by the help of what he does not understand. The morbid logician seeks to make everything lucid, and succeeds in making everything mysterious. The mystic allows one thing to be mysterious, and everything else becomes lucid. (p49)
Buddhism is centripetal, but Christianity is centrifugal: it breaks out. For the circle is perfect and infinite in its nature; but it is fixed forever in its size; it can never be larger or smaller. But the cross, though it has at its heart a collision and a contradiction, can extend its four arms for ever without altering its shape. Because it has a paradox at its centre it can grow without changing. The circle returns upon itself and is bound. The cross opens its arms to the four winds; it is a signpost for free travellers.
Symbols alone are of even a cloudy value in speaking of this deep matter; and another symbol from physical nature will express sufficiently well the real place of mysticism before mankind. The one created thing which we cannot look at is the one thing in the light of which we look at everything. Like the sun at noonday, mysticism explains everything else by the blaze of its own victorious invisibility. Detached intellectualism is (in the exact sense of a popular phrase) all moonshine; for it is light without heat, and it is secondary light, reflected from a dead world. But the Greeks were right when they made Apollo the god of imagination and of sanity; for he was both the patron of poetry and the patron of healing. Of necessary dogmas and a special creed I shall speak later. But that transcendentalism by which all men live has primarily much the position of the sun in the sky. We are conscious of it as of a kind of splendid confusion; it is something both shining and shapeless, at once a blaze and a blur. But the circle of the moon is as clear and unmistakable, as recurrent and inevitable, as the circle of Euclid on a blackboard. For the moon is utterly reasonable; and the moon is the mother of lunatics and has given to them her name. (p50-51)
But what we suffer from to-day is humility in the wrong place. Modesty has moved from the organ of ambition. Modesty has settled upon the organ of conviction; where it was never meant to be. A man was meant to be doubtful about himself, but undoubting about the truth; this has been exactly reversed. Nowadays the part of a man that a man does assert is exactly the part he ought not to assert - himself. The part he doubts is exactly the part he ought not to doubt - Divine Reason. Huxley preached a humility content to learn from Nature. But the new sceptic is so humble that he doubts if he can ever learn. Thus we should be wrong if we had said hastily that there is no humility typical of our time. The truth is that there is a real humility typical of our time; but it so happens that it is practically a more poisonous humility than the wildest prostrations of the ascetic. The old humility was a spur that prevented a man from stopping; not a nail in his boot that prevented him from going on. For the old humility made a man doubtful about his efforts, which might make him work harder. But the new humility makes a man doubtful about his aims, which will make him stop from working altogether. (p55-56)
See also - G.K. Chesterton - The Everlasting Man.