1. Are the optics of your scope good quality. Bad optics will never focus sharply and image contrast will always be poor. In a reflecting telescope if the mirror is homemade, has it been tested so that you know if it can perform well?
2. Collimation, (optical alignment) could be the problem. Visit this site on how to fix this and many other telescope problems. http://astronomy.trilobytes.com.au/scope/fix-it.htm
3. Bad seeing (turbulence). If Mars is low in the sky, atmospheric turbulence can obliterate contrast and detail on the planet. Wait for it to gain altitude say greater than 30 degrees. Sometimes seeing may be bad even when Mars is high. It depends on atmospheric conditions at the time.
4. Speaking of turbulence. Thermal currents inside the telescope tube will have a similar effect if the scope is transported from a warm interior to cold outdoors. Allow the scope 1/2hr to 1hr to reach thermal equilibrium outdoors before beginning to search for planetary detail.
5. Even in an 8” with good optics and good seeing conditions the dusky markings on the planet require some patience and practice to discern especially if you are new to observing. The bright polar cap however should be easily visible.
6. In a 3.5“ Maksutov, The polar cap is always visible and the most prominent markings (Sinus Sabius, Syrtis Major, Mare Sirenum, Mare Cimmerium and the dark hood around the ever shrinking polar cap) are identifiable most of the time. They are better with a homemade 6” Newtonian.
7. Sometimes a larger aperture performs worse than a smaller aperture under bad seeing conditions. You may have been caught out by this. It depends on the prevailing general seeing conditions in your locality which may be determined by season, adjacent buildings, bodies of water, vegetation and other such factors. You might like to try a 4“ or 6” aperture mask over the top end of the scope to see if this beats the seeing conditions. It might also cure a badly figured homemade mirror if that is the problem.
8. A red or orange filter will help if the seeing is good to begin with. Otherwise it makes little difference. On a recent occasion an orange filter improved the contrast on Mars in the 3.5“ Maksutov but made no difference in the 6”. A filter however, even if it does nothing else, can reduce the “glariness” of the planet so that it is easier on the eye.
9. Provided the optics in your scope are good to begin with, keep on persisting with the above points in mind and you are sure to get a good night when Mars will deliver! As a rule, with good optics, under good seeing conditions, a larger aperture will always outperform a smaller one.