The original meaning of the word hash was "to hack, chop into small pieces" from the French hacher, from Old French hache meaning axe. It often had culinary use, a hash being a chopped up mixture of things, most notably the dish hash browns, made of shredded potatoes. The expression to make hash out of sth/sbd, means to ruin sth/sbd and this follows logically from chop into small pieces
There is a derivation rehash, that dates from 1822, meaning talk over, discuss again or present in another form.
Later, hash over appears and although I cannot find a precise date, it seems to appear before hash out. The meaning is somewhat similar to rehash. A similar expression thrash out, had already been used for some time to mean working through the details of something and derived from the threshing of cereal crops to separate out the grain.
From around the 1940s I found some uses in google books of the form to hash out problems/differences, meaning to settle an argument through extensive debate. It seems a natural progression given the other expressions presented here. In modern usage hashing out plans/ideasstill implies vigorous discussion but isn't necessarily an argument.