A Value Function in RL estimates the expected long-term return starting from a state (or state-action pair). It guides an agent on which states are more valuable to occupy or which actions to take. Methods like temporal-difference learning update the value function based on rewards and next state values.