Thank you all for your input and help on this. We had a discussion with them on Friday night, and it was a little rocky to say the least. The husband you could definitely tell was getting fired up at points, and the wife needed to calm him down. I see where they were coming from as they have lived there for 12 years with no issues, but all of this was even done prior to them living there. I pointed that out to them a couple times but it was hard to get through to them. He kept saying 'according to the sight line in the concrete, this is our property.' Then I would say well according to your theory on your sight line, your mailbox is on our property. I never wanted to bring up the mailbox, but it was an issue for me from day 1, but I am not one for confrontation. He even mentioned at one time 'You knew the RV pad was small when you moved in, and you still decided to move in.' Right, we were first time home buyers, and didn't think we needed to measure the RV pad, or anything. At the time I was like I can park my trailer there, that will be great! I knew it wouldn't fit anything big like a toy hauler, but at least being able to run my small trailer up the side of the house will be nice. Figured I could put the eventual toy hauler where it currently is, no biggie. In fact if you look at the photo Nate posted, you can see my trailer sitting in front of that area because there was a chain link fence that was on my side that I had to remove to get it back there. He then mentioned that he is meticulous with his yard, and tired of all our leaves. I wanted to be like, "well, the trees were they prior to you moving in, didn't you know you would possibly get leaves?" But I wanted to keep it from getting worse. Come to find out, her sister lived in our house, and was the one that planted all the trees! My wife likes them, but I would like to get rid of at least 3-4 of them.
My wife mentioned to me he was completely different as far as communication wise compared to earlier in the day. One thing that did come up however is that he said they could get a written letter from the previous owner that the backyard was the new established line, and according to the surveyors we talked too, that would hold up compared to a 30 year old plat map.
Anyway, after the discussion it was decided that they would close in the fence next to their house (really we were not too worried about it - they did tell me verbally they would be doing that when we discussed removing that chain link) and remove their mailbox. They said they would think on the smaller fence and let us know the next day.
We did (we think) find the original property line marks in the concrete curbing, and if they were indeed the correct ones, even measurement wise they were not 100% accurate, but did support our theory of being short on both sides. The wife texted my wife and asked if we could come over and talk about the fence. The husband wasn't in the same room, but I am sure was home. She mentioned that it sounded like the biggest complaint we had was the mailbox, and not being able to access our trailer, which is true. She mentioned they didn't sleep most of the night trying to figure out a solution. She asked if we would be ok with only going half way down the driveway. Long story short and lots of back and forth, my wife and I decided just to agree with it, and move on. Decided we would rather have our integrity, not burn bridges and make everything awkward for the 10-15 years we plan to be at this house, and still have the convenience of daycare rather than a foot or two of additional property. Not including the survey, possibility of court fees, and other things that we just do not have the time nor interest in fighting. Our neighbor to the east was like "there is room to park both of those trailers next to the houses, just share it!" Sometimes I wish other people would mention that to them as well, but really it isn't between them. If they feel like they are doing the right thing, I guess that is on them.
The survey would run about $1200-2300. That is to do the full survey, record it with the county, file everything, etc. That is the option we would have needed to even consider fighting this. We knew we would lose the backyard as per the earlier conversation and verbal agreement with the old home owner, but were more worried about this new fence going up.
In the end, we are a bit bitter about it, but just decided it wasn't worth fighting. We don't know what the right decision would have been, or is, but we just decided it's not worth the headache, hassle, added stress or anxiety. The possibility of all the fees and charges would add up to more than our trailer cost, and at what point when do you draw the line? I just hope I don't hit their fence when I go to back up the trailer from here on out.