You said you test every swatch on the wall first, which is smart. How long do you leave them up before you decide? I made a mistake once by picking a color right away, but it looked totally different after it dried fully and in the evening light.
Last year, I picked a color called 'Sea Mist' that looked perfect on the tiny card. I was so confident I skipped testing and painted the whole hallway. It dried into this weird hospital green that made me feel queasy just walking past. My buddy came over and asked if I was trying to match the color of old broccoli. Now I tape up those samples and stare at them for days like they're going to tell me the meaning of life.
You are so right about the evening light, that's the real test. I leave mine up for at least three days now to see them in all different weather and light. I once picked a gray that looked perfect at noon, but by sunset it had this weird blue tone that made the whole room feel cold. Watching the color change over a full day is the only way to know for sure.
Why rush such a big choice? I test swatches for a full week, no less. @taylor_park68 nailed it with the evening light thing. I learned from a yellow that turned gross in my kitchen light. Now I watch colors at all times of day. Taking days to decide stops awful mistakes.