Nope, you should only worry when you're 28 weeks and up. Your baby has a lot of room in there, so you may not feel him/her move even though they're moving.
No need to worry. Your baby still has a lot of room in there and is still small so it’s normal not to feel any movement for a couple of days. My baby did the same thing when I was between 21-23 weeks and it always worried me, but my midwife always assured me that it was fine and perfectly normal.