I would think that the automotive type cable would be a larger gauge than the wire from your TM to your breaker.
Just so that we are clear, Your TM has two wires positive and negative. We will call them P and N.
P should run to one side of your breaker. N should run to a negative post on one of your batteries.
From the other end of your breaker, a wire should run to the positive side of one of your batteries. I would want this wire to be the same as the P wire in terms of gauge.
Others that are far more experienced than me might chime in here. I would look at all the connections and clean them if possible. That breaker shouldn't pop from heat. It should pop because the circuit is overloaded.
Might, and I say might, be a bad breaker too. Many a times we look over the simplist of things. Has it always done this with this setup or has it just started? If it has recently started, I would lean towards coroaded wires or a bad breaker! Hope that helps.
I have OCD "Obsessive Crappie Disorder"