Olives botanically are drupes, a type of plants’ fruit. This is because they are the part of the plant that contains the seed and the singular seed has a hard shell. This is botanically the same as fruits like peaches. However, one likely would not consider olives to be very similar to peaches. This is because botanical definitions are not very useful for most people.
Many people who don’t live near the Mediterranean may not have ever even seen a raw olive. We typically eat them brined, with their seeds removed, and/or otherwise processed. This is indicative of being a vegetable. In addition, olives have zero sugar; they’re very bitter. Even after processing, they’re still a savory flavor. I don’t think anyone eats olives like a fruit. They’re always processed and added to savory dishes like pizza and pasta. This makes them vegetables.
The US Court of Customs Appeals has called olives fruit several times. Its reasoning was that they are botanically the fruit of the plant and that that makes it unable to be a vegetable. This dichotomy is logically incorrect however. In reality, something can be both a “fruit” botanically and a vegetable.