I have a trace inverter 2512. The batteries are all brand new the four house batteries and the 2 starting batteries are also new. I have several problems. All I think related to 12 volt. Sometimes I will have absolutely no house battery power at all and then it will mysteriously come on. Sometimes my house batteries will charge sometimes they will not. Sometimes my inverter will invert sometimes it will not. The house batteries will never charge off my alternator with only the engine running. I replaced the device that splits the aternator power between the two types of batteries. I found that this device was defective last fall. (the large blue device with three poles going to it center pole is from alternator and two other poles to each bank of batteries.) My engine batteries were not charging prior to replacing this. I think my battery tender the green thing with the green led light works because sometimes it will come on and this is the only time my house batteries will charge. What I have found is that when my house batteries charge I have a ground from the house batteries too the frame. When nothing is charging or working I have no ground. I checked this by testing continuity between my house battery negative cable to the frame. It is my understanding that the house battery ground goes from the battery bank directly to the inverter-through a shunt in the inverter- then is grounded to the chassis. It is almost as if I am not getting a ground from the inverter or if so it is intermittant (that is the next thing I am checking although with a casual glance it seems fine). My house batteries will not charge when I am hooked to shore power or when the generator is working. It almost appears to me that I have a bad ground from the inverter to the frame or something is wrong internally in my inverter. It is odd that it is so intermittant. It is a 99 Holiday rambler navigator with the 10.9 liter and the aqua hot system if that makes any difference. Actually it will charge sometimes mostly after I just start the generator or plug it in. I will see the green led light come on the little green box and I test the batteries and they are charging but, then it goes off and seems to lose its ground. When the green light comes on I have a ground if that makes any sense.
I think you're on the right track with suspecting the ground to be a problem. You'll need to pull off, clean, and reconnect each ground and negative starting with the house batteries. It seems the "supply" of 12V coming from the batteries is getting interrupted some where in that path. Disconnect from shore power when you do this and have a fully charged set of batteries too. Use a portable charger to charge them up for now.
I don't think the house battery shunt is hooked up the way you think it is. If the Inverter/Charger and remote control monitor panel was factory installed, then I suspect the shunt is internal to the inverter and only allows measurement of current provided to, or drawn from, your house batteries by the Inverter/Charger. The shunt would effectively be in the Inverter/charger negative path, but not the battery negative.
Either the I/C or the batteries has to be grounded directly to the frame to make the measurements via the shunt function properly.
If you have a Trimetric or a Xantrex Link 10/20 (installed after market) then I would agree with you that the shunt is very likely in your house battery negative cabling.
I think what you have is a defective negative connection from your house batteries to frame ground. Remove, inspect, clean and tighten all negative connections from batteries to frame and your problem will likely disappear.
Measure voltage at (exactly and precisely) the battery connection STUDS (not the nuts, not the wire lugs) with inverter off, and then with the inverter running under either a great load or great charging protocol. What voltages do you see at the studs?