Libertarianism is often labeled as a philosophy of selfishness. This is a mistake, and to see why it is a mistake, we need to review the basic principles of libertarianism.
Liberalism rests on the concepts of self-ownership, homesteading, and non-aggression. According to this theory, each person has the right to be the master of his or her own body. This right stems from the belief that each person has free will. The existence of free will is understood as an axiom--a self-evident truth that cannot be proven or disproven.
The next step in the logic of the libertarian theory of rights is mixing one's labour with nature--homesteading. One acquires legitimate ownership over the products of his or her own labour when these products are created by using resources that were not previously owned by anyone. In other words, natural resources belong to the one who finds them first. In cases when a resource is already owned by someone, another person can legitimately acquire this resource only through a voluntary exchange.
The key feature of a voluntary exchange is that the exchange is performed in the absence of physical aggression or a threat of physical aggression. This means that it is illegitimate to take one's resources without one's consent, and it is critical that this consent is obtained in the absence of a threat of physical aggression.
From this, we can see that the libertarian theory requires one only to respect the property of others. It does not, however, dictate what kind of preferences one should have when it comes to using his own property. For example, libertarianism does not state whether one should use his own property for charitable purposes, or to make a profit, or keep it all locked up at all times. Libertarianism only opposes taking one's property without his consent, nothing more and nothing less than that.
I have more to say on the problems of using subjective judgment of others' actions (i.e., "action x is a selfish thing to do") as a normative criterion, but this is unrelated to libertarianism, so let's leave that for another occasion. Until then, you might want to take a look at this piece, where I explore the concept of intellectual property in a similar context.