heartbeat logmessage fix

[Bowrna]

closes: #31810
relates: #31810

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[Bowrna]

Some of the suggestions provided in the issue by @hterik

A: Don't log the whole stack trace, instead just log this as an error message.
B: Include a descriptive message of the consequence of this error in the log.
C: When heartbeat recovers, log it.
D: If heartbeat was recovered after scheduler_health_check_threshold, log another more severe error message including descriptive message of the consequences.

the current changes cover A, B and D. I will figure out how I can find if the heartbeat is recovering and finish C part.

[Bowrna]

@kaxil @ashb @XD-DENG Please review this PR and let me know about your views.

[hterik]

As a user of airflow reading the logs of a dag i would not understand what this means to me. Is this something i have to react to? Do i need to contact my admins? Is the dag results corrupted? Should i restart the scheduler?

This error isn't necessarily a problem with the scheduler. More often it is a problem of the executor not being able to reach the database, due to transient network problems. As long as this error is transient and recovers shortly, the consequence of this is usually none. The log message should reflect this. If this is too much to fit into a log-message, linking to the architecture documentation at airflow.apache.org as suggested by potiuk above sounds like a good proposal.

[Bowrna]

@potiuk In this PR, I am specifically working on the heartbeat failure when updating the table about the heartbeat timestamp. So its more related to db problems when executor wants to update the table. But with above comments, I think the scope for this issue is expanding. Is my understanding right @potiuk ?

[potiuk]

@potiuk In this PR, I am specifically working on the heartbeat failure when updating the table about the heartbeat timestamp. So its more related to db problems when executor wants to update the table. But with above comments, I think the scope for this issue is expanding. Is my understanding right @potiuk ?

That's the proposal - but it's just suggestion, if you want to stay with small change - up to you.

[Bowrna]

@potiuk thanks for the update. Let me break the proposal into a few more tasks and work on it.

[potiuk]

Suggested change

# self.log.exception("%s heartbeat got an exception", self.__class__.__name__)

[potiuk]

LGTM: with the small NIT of removing the commented out line

[Bowrna]

thanks @potiuk let me fix it

[Bowrna]

I wrongly did a git force push with main branch changes. Please excuse. I will clean up and commit the changes