Ardor

Rectified Linear Unit (ReLU)

ReLU is an activation function defined as max(0, x). It solves the vanishing gradient issue of older functions like sigmoid by not saturating for positive values. However, it can cause “dying ReLUs” when inputs stay negative, leading to zero gradients.

Still doing it by hand? Describe it once and let it run.