Yes. This is done through the Feed Settings section of the PeepSo Configuration Settings (PeepSo → Configuration → CM Answers).

The Post Template can include some shortcodes to make the post content nice and relevant to the question.

To learn more about customizing activity posts see this Help article.