Do I need to travel to Poland to set up the company?

Not usually. The company can be formed remotely using powers of attorney and the S24 online system. A visit may be helpful, and sometimes required, when opening the bank account.