Yes. This is done through the Feed Settings section of the PeepSo Configuration Settings (PeepSo → Configuration → CM Answers).
The Post Template can include some shortcodes to make the post content nice and relevant to the question.
To learn more about customizing activity posts see this Help article.