Head coverings for women are mentioned in the Torah in chapter 5 of Numbers, in something called the Sotah ritual. The Sotah was a woman suspected of adultery, and the ritual was a test. Part of that involved uncovering her hair, so the rabbis deduced from those verses that there was a biblical commandment for married women to cover their hair. Elsewhere in Tanach, in Song of Songs, there is the suggestion that the sight of women’s hair is erotic, from which the rabbis determined that hatless women would be distracting to a man at prayer.
The specifics of hair covering (how much cover, and when) was a matter of communal custom in ancient times, and it remains so today.
Today in Reform and Conservative synagogues women are welcome to cover their heads for prayer if they wish to do so; in some congregations, it is a requirement. Usually if there is a rule about it, it will be posted outside the sanctuary, and coverings of some sort will be available. In a Reform or Conservative shul, the kippah or yarmulke has become a common sight on men or women. These days it is not a modesty issue, but a matter of respect for the activity of prayer and the awareness of the Divine.
Personally, putting on a head covering is part of my routine for prayer and study. It’s a way of telling my body, “OK, time to get serious now!”
There is a wider variety of practice among Orthodox congregations. There, a kippah may be seen as a men’s garment, and therefore is not worn by women. The lace hats you described in your original question are a feminized version of the kippah. Women may wear a tichel (head scarf, pictured above) or a regular hat, or in some communities they may cover their own hair with a sheitel (wig.) If you visit an Orthodox synagogue for services, wear a scarf; that will usually be sufficient for guests.
For a wonderful article on the subject, read “Hair Coverings for Married Women” by Alieza Salzberg.