The plaintiff shall only be entitled to recover damages for withholding the property for the term of six years next preceding the commencement of the action, and for any period that may elapse from such commencement, to the time of giving a verdict therein, exclusive of the use of permanent improvements made by the defendant. When permanent improvements have been made upon the property by the defendant, or those under whom he or she claims holding under color of title adversely to the claim of the plaintiff, in good faith, the value thereof at the time of trial shall be allowed as a setoff against such damages.