According to the CDC, you can use tap water and soap to wash hands during a boil water advisory. Be sure to scrub your hands with soap and water for at least 20 seconds. Then, rinse them well under running water. If soap and water are not available, use an alcohol-based hand sanitizer that contains at least 60% alcohol.
You can also shower or bathe, but be careful not to swallow any water. Use caution when bathing babies and young children. Consider giving them a sponge bath to reduce the chance of them swallowing water.