DAOS-18427 control: Retry system self-heal eval

[tanabarr]

Retry dmg self-heal eval command when engine not started error is
returned. Do this by updating the IsUnavailable() helper.

Features: control

Steps for the author:

• Commit message follows the guidelines.
• Appropriate Features or Test-tag pragmas were used.
• Appropriate Functional Test Stages were run.
• At least two positive code reviews including at least one code owner from each category referenced in the PR.
• Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

• Gatekeeper requested (daos-gatekeeper added as a reviewer).

[github-actions]

Ticket title is 'dmg system self-heal eval sometimes fails when a rank is stopped'
Status is 'In Review'
Labels: 'test_2.8'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-18427

[daltonbohning]

I verified this resolves the issue I was seeing in #17353

[kjacque]

Not a change request now, but I think it would be good to generalize this in the future. We have different retryTestFns for a lot of commands where I think the circumstances under which we'd retry are largely the same. Couldn't a lot of MS commands be similarly affected by a leadership change?

[tanabarr]

@mjmac may be able to provide guidance for a future change with regard to this. @kjacque can you approve this PR unless you have blocking requests please?

[kjacque]

This seems reasonable to me, but @mjmac may have more context on why this function was set up this way to begin with, checking the strings instead of comparing errors. If the assumption is that server-side errors got flattened somehow, would we actually get the error in the form of a Fault? Or just a string representation of a Fault?

[tanabarr]

This change works in that it addresses the issue and it makes sense that the other errors are defined in this file as sentinel errors rather than faults. Please approve unless there is anything else blocking.