There are a few reasons why you should protest the shoplifting, but they each have different ramifications:
1) Hocheach Tochiach es Amisecha: You are obligated to rebuke your fellow when he contravenes the Torah laws. Regarding something which is “mefurash baTorah”, we don’t say מוטב שיהיו שוגגים ואל יהיו מזידים, and you must rebuke him, even if you think it likely that he will not listen. This would obligate you to rebuke him privately.
2) Chillul Hashem – I think the earliest source for this is Rabbeinu Yonah, and it is brought by the Mishna Berura – you must speak out when someone publicly transgresses the Torah, so that it doesn’t look like you consent to what he does. This would obligate a public comment. However, if nobody else saw this, you cause more Chillul Hashem by saying anything publicly than by ignoring it.
3) Lehafrish Me’Issurim – You are obligated to stop a fellow Jew from transgressing any Issur. You may/must use necessary physical force to enforce that nobody transgresses a commandment (BB”K 28a). There is a discussion in the Poskim if this obligation extends to enforce positive commandments or only to prevent transgression of negative commandments (originally Ketzos v. Nesivos 3). If you have the capacity to stop him stealing, you must definitely do so. However, the moment he picks it up with intention to take it, he has already transgressed the prohibition; now there is only a chiyuv to return it.
I am not sure what the halachah would be regarding the obligation to return a stolen item – would that be classified under positive commandment (in which case it would be a machlokes Ketzos and Nesivos) or is it the same as a negative commandment because it is a תיקון of לא תגזול in the form of לאו הניתק לעשה.
Also, if the owner is a Goy, then although it is prohibited to steal, there is no obligation to return the item once you have taken it. Therefore, once he picked it up, he is allowed to keep it, unless there is an issue of חילול ה’ (which there won’t be if you are the only guy who has seen it and don’t plan on trumpeting it).
The Halachos of Macha’ah (which cases require a protest, what constitutes a protest, whether to protest publicly or privately) are discussed by the Mishna Berura in the beginning of Hilchos Yom Kippur. If I remember correctly, he is not consistent with his policy in Chafetz Chaim (question mark).