I am sorry, I did not see this post till just now.
Yes, there is ambiguity with subjunctives, and I am not sure I can make anything more clear than how you put it in your question.
"If I visited here before, I do not remember." If I visited is in subjunctive, yes, I would say so because the person is holding it out as a possibility that may or may not have been real. In that case the grammatical form does not change, since we're in the past tense, and you would have to be in Latin or Greek to see the difference in form, so in some ways it doesn't matter in the mechanical production of the sentence, though it would matter in the parsing of the sentence.
"If I were rich, I would contribute more to your cause." This one is more pertinent, perhaps, since the verb does change or could change, since I could write "If I was rich...".
My honest answer is that subjunctive use is not that clear in English, and with the gradual change of language over time, I would argue that people have all but abandoned the subjunctive.
I will ask my daughter who is majoring in Attic Greek if she can shed more light on issues of subjunctive use as they relate to English and Greek use of it, to see if I can elaborate on this, but so far, I am not sure I can add more to what you asked.