three posts about how we can deal with being different levels of rational at different times, or how we can (and should!) be ok with having heuristics while not being ruled by them: one two three.
(takes a long time, I totally understand if you want to just tune out now)

Something like the model in the last post makes a lot of sense:
explicitly model system
feel emotion
endorse value based on essence (reified or not)

As we learn more, we move up the ladder, going from “well, there is this magical thing called ‘justice’ and it is good” to “I don’t know, I just feel bad when you wrong me, and somewhat better when you are punished for it” to “here’s how we can most effectively prevent crimes.”

Or, “fruits and vegetables are healthy” -> “I feel better when I eat them” -> “I need all these 1000 vitamins and minerals in just these proportions, so eating these foods will satisfy them.”

(Notice that we’re not, in the “healthiness” case, actually fully successfully at the “explicitly model system” phase yet! Soylent tried, but they’re idiots.)

I think it’s straightforward to say that we should strive to be as far up the ladder as we can be, while being humble about where we are on the ladder.

(Also, most arguments about politics or whatever would be solved if you’re just more specific.)

I’ll hold off on the question of “do people have real value differences?” because I don’t know. I was just sitting on these and wanted to post something about them, but don’t have a fully formed thought yet.