>>It was just the only food deemed cheap enough to feed the poor.

It hadn't been accepted as part of the diet, so from the English view, it might have been a matter of "let them eat dirt". But I think the key factor was that, unlike grain, it didn't need to be harvested at the end of the summer. Grain left in place would rot. So, it needed to be stored, and was subject to export by landowners and any privately grown grain subject to appropriation.

The potato, meanwhile, was something the English did not want. On top of that, it could be left in the ground, only harvested as needed. Those two together meant it was something the Irish could grow and expect to bring to table.

Agreed on the poor being seen as expendable, particularly Irish poor. It's not clear to me they were even seen as 'people' (as evidence I'd offer Swift's "A Modest Proposal").