The early church also worshiped on Pentecost, so why don't you keep Pentecost as well?
The New Testament mentions that the church started on the day of Pentecost; however, it does not mention that the church celebrated Pentecost or observed any of the regulations for Pentecost. It simply marked the time when the church began by noting that the day was Pentecost. "When the Day of Pentecost had fully come, they were all with one accord in one place" (Acts 2:1).
Pentecost always fell 50 days after the feast of the Unleaven Bread. Since this feast always ended on a Sabbath day (the seventh day of the week), it is noted that the church began on the first day of the week. Because the church worships on the first day of the week, we do happen to worship on Pentecost -- not because we celebrate Pentecost, but because the times of the two events happen to coincide.