In the famous 1890 paper by Warren and Brandeis, The Right to Privacy, the authors noted that there was no Constitutional right to privacy. They noted:
That the individual shall have full protection in person and in property is a principle as old as the common law; but it has been found necessary from time to time to define anew the exact nature and extent of such protection. Political, social, and economic changes entail the recognition of new rights, and the common law, in its eternal youth, grows to meet the demands of society. Thus, in very early times, the law gave a remedy only for physical interference with life and property, for trespasses vi et armis. Then the "right to life" served only to protect the subject from battery in its various forms; liberty meant freedom from actual restraint; and the right to property secured to the individual his lands and his cattle. Later, there came a recognition of man's spiritual nature, of his feelings and his intellect. Gradually the scope of these legal rights broadened; and now the right to life has come to mean the right to enjoy life--the right to be let alone, the right to liberty secures the exercise of extensive civil privileges; and the term "property" has grown to comprise every form of possession-- intangible, as well as tangible.
They continue:
It is our purpose to consider whether the existing law affords a principle which can properly be invoked to protect the privacy of the individual; and, if it does, what the nature and extent of such protection is.
Owing to the nature of the instruments by which privacy is invaded, the injury inflicted bears a superficial resemblance to the wrongs dealt with by the law of slander and of libel, while a legal remedy for such injury seems to involve the treatment of mere wounded feelings, as a substantive cause of action. The principle on which the law of defamation rests, covers, however, a radically different class of effects from those for which attention is now asked. It deals only with damage to reputation, with the injury done to the individual in his external relations to the community, by lowering him in the estimation of his fellows. The matter published of him, however widely circulated, and however unsuited to publicity, must, in order to be actionable, have a direct tendency to injure him in his intercourse with others, and even if in writing or in print, must subject him to the hatred, ridicule, or contempt of his fellow men--the effect of the publication upon his estimate of himself and upon his own feelings not forming an essential element in the cause of action. In short, the wrongs and correlative rights recognized by the law of slander and libel are in their nature material rather than spiritual. That branch of the law simply extends the protection surrounding physical property to certain of the conditions necessary or helpful to worldly prosperity. On the other hand, our law recognizes no principle upon which compensation can be granted for mere injury to the feelings. However painful the mental effects upon another of an act, though purely wanton or even malicious, yet if the act itself is otherwise lawful, the suffering inflicted is damnum absque injuria. Injury of feelings may indeed be taken account of, in ascertaining the amount of damages when attending what is recognized as a legal injury; but our system, unlike the Roman law, does not afford a remedy even for mental suffering which results from mere contumely and insult, from an intentional and unwarranted violation of the "honor" of another
The conclusion is that they try to find one in Common Law and Torts, not in the Constitution. Thus the issue of privacy has at best been accomplishe by law not by the Constitution.