CFEngine Scheduling Tasks

[Rebecca-Dieter-Teamwire]

Hi,

we are having some interesting behavior with some schedules within our promises.

In one promise we are using the scheduler to execute a script at certain full hours.
First example:

      Hr00.Min00|Hr08.Min00|Hr13.Min00|Hr18.Min00::
        "/usr/local/src/example.sh"
          inform  => "false",
          classes => if_else("successful","broken");

The script seems to finish within a minute

In the second promises we are using the scheduler to execute commands
Second Example:

    (Hr08.Min00_05)|(Hr13.Min00_05)|(Hr18.Min00_05)::
      "mysqldump example_db > example.dest"
        inform  => "true",
        contain => in_shell,
        classes => if_notkept("broken");

      "find example -type f -not -newermt '-48 hours' -exec rm {} \;"
        inform  => "true",
        contain => in_shell;

This currently takes about 5 minutes

When we used the same scheduling as in the first example within the second Example so:

      Hr08.Min00|Hr13.Min00|Hr18.Min00::
      "mysqldump example_db > example.dest"
        inform  => "true",
        contain => in_shell,
        [...]

This didn't workout as it should.

Is this cause the scheduler only runs within the first minute of the hour and the commands took to long (over a minute)?
Or does any one have an idea why this behavior is shown?

Thanks in advance :)

[larsewi]

Hi @Rebecca-Dieter-Teamwire :)

In your example, you are using time-based classes. These are so-called hard classes, meaning that they are set when the agent wakes up. More specifically, time-based classes are set depending on the system time. This means that the class Min00 is only set if the agent happens to start executing within the first minute of an hour, which is quite unlikely.

Using the following command, you can see what classes are set:

$ sudo cf-agent --show-evaluated-classes="Min"
Class name                                                   Meta tags                                Comment                                 
GMT_Min25_30                                                 time_based,cfengine_internal_time_based_autoremove,source=agent,hardclass                                         
GMT_Min26                                                    time_based,cfengine_internal_time_based_autoremove,source=agent,hardclass                                         
Min25_30                                                     time_based,cfengine_internal_time_based_autoremove,source=agent,hardclass                                         
Min26                                                        time_based,cfengine_internal_time_based_autoremove,source=agent,hardclass 

By default, the agent runs once every 5 minutes, but this can be configured in cf-execd agent execution schedule. Thus, if you want to make sure the promise will be executed, you should probably use the Min00_05.

Hope this helps 😄

[nickanderson]

I'll chime in just to highlight splaytime (https://docs.cfengine.com/docs/3.21/reference-components-cf-execd.html#splaytime). splaytime is the maximum number of minutes that cf-execd should delay execution of cf-agent. The actual time is calculated based on a hash of the hostname so that hosts on the same schedule will not all wake up and execute policy simultaneously. By default this is 4 minutes, which should leave a 1 minute quiet period where new agent's aren't started.

CFEngine hard classes and sys vars are resolved during agent initialization. So, it's the classes that are set during initialization that will be set for the run, classes are not updated every minute during an agent run.

This snippet should help to illustrate that:

bundle agent __main__
{

  commands:
    "/bin/sleep 500;# GRAB"
      handle => "commands_sleep",
        contain => in_shell,
      depends_on => { "report_date_1" };

  reports:
    "1) $(sys.date); # GRAB"
      handle => "report_date_1";

    "2) $(sys.date); # GRAB"
      depends_on => { "commands_sleep" };

      "3) Time classes: $(with); #GRAB"
        with => join( ", ", classesmatching( ".*", "time_based" ) ),
        depends_on => { "commands_sleep" };
}

Running the policy, you can see that the agent started around 9:27 and both before and after a 500s sleep sys.date remained 09:27. Similarly classes with the time_based tag reflect the agent start time, not the current time within execution.

$ cf-agent -Kvlf /tmp/example.cf | grep GRAB
2024-03-25T09:27:18-0500  verbose: Skipping promise '/bin/sleep 500;# GRAB', as promise dependency 'report_date_1' has not yet been kept
2024-03-25T09:27:18-0500  verbose: Skipping promise '2) Mon Mar 25 09:27:18 2024; # GRAB', as promise dependency 'commands_sleep' has not yet been kept
2024-03-25T09:27:18-0500  verbose: Skipping promise '3) Time classes: GMT_Min27, Monday, Q2, GMT_Monday, Day25, GMT_Afternoon, Min25_30, Yr2024, GMT_Hr14_Q2, GMT_Hr14, GMT_Yr2024, Min27, GMT_Lcycle_2, Hr09, March, Hr9, Hr09_Q2, Lcycle_2, GMT_Min25_30, Morning, GMT_Q2, GMT_March, GMT_Day25; #GRAB', as promise dependency 'commands_sleep' has not yet been kept
2024-03-25T09:27:18-0500  verbose: Skipping promise '/bin/sleep 500;# GRAB', as promise dependency 'report_date_1' has not yet been kept
2024-03-25T09:27:18-0500  verbose: P:    Promiser/affected object: '1) Mon Mar 25 09:27:18 2024; . GRAB'
2024-03-25T09:27:18-0500  verbose: P:    Stack path: /default/main/reports/'1) Mon Mar 25 09:27:18 2024; . GRAB'[1]
R: 1) Mon Mar 25 09:27:18 2024; # GRAB
2024-03-25T09:27:18-0500  verbose: Skipping promise '2) Mon Mar 25 09:27:18 2024; # GRAB', as promise dependency 'commands_sleep' has not yet been kept
2024-03-25T09:27:18-0500  verbose: Skipping promise '3) Time classes: GMT_Min27, Monday, Q2, GMT_Monday, Day25, GMT_Afternoon, Min25_30, Yr2024, GMT_Hr14_Q2, GMT_Hr14, GMT_Yr2024, Min27, GMT_Lcycle_2, Hr09, March, Hr9, Hr09_Q2, Lcycle_2, GMT_Min25_30, Morning, GMT_Q2, GMT_March, GMT_Day25; #GRAB', as promise dependency 'commands_sleep' has not yet been kept
2024-03-25T09:27:18-0500  verbose: P:    Promiser/affected object: '/bin/sleep 500;. GRAB'
2024-03-25T09:27:18-0500  verbose: P:    Stack path: /default/main/commands/'/bin/sleep 500;. GRAB'[1]
2024-03-25T09:27:18-0500     info: Executing 'no timeout' ... '/bin/sleep 500;# GRAB'
2024-03-25T09:35:38-0500  verbose: Finished command related to promiser '/bin/sleep 500;# GRAB' -- succeeded
2024-03-25T09:35:38-0500     info: Completed execution of '/bin/sleep 500;# GRAB'
2024-03-25T09:35:38-0500  verbose: P: END commands promise (/bin/sleep 500;# GRAB)
2024-03-25T09:35:38-0500  verbose: P:    Promiser/affected object: '2) Mon Mar 25 09:27:18 2024; . GRAB'
2024-03-25T09:35:38-0500  verbose: P:    Stack path: /default/main/reports/'2) Mon Mar 25 09:27:18 2024; . GRAB'[1]
R: 2) Mon Mar 25 09:27:18 2024; # GRAB
2024-03-25T09:35:38-0500  verbose: P:    Stack path: /default/main/reports/'3) Time classes: GMT_Min27, Monday, Q2, GMT_Monday, Day25, GMT_Afternoon, Min25_30, Yr2024, GMT_Hr14_Q2, GMT_Hr14, GMT_Yr2024, Min27, GMT_Lcycle_2, Hr09, March, Hr9, Hr09_Q2, Lcycle_2, GMT_Min25_30, Morning, GMT_Q2, GMT_March, GMT_Day25; .GRAB'[1]
R: 3) Time classes: GMT_Min27, Monday, Q2, GMT_Monday, Day25, GMT_Afternoon, Min25_30, Yr2024, GMT_Hr14_Q2, GMT_Hr14, GMT_Yr2024, Min27, GMT_Lcycle_2, Hr09, March, Hr9, Hr09_Q2, Lcycle_2, GMT_Min25_30, Morning, GMT_Q2, GMT_March, GMT_Day25; #GRAB

Also, you may want to consider having CFEngine schedule long running tasks (cron, at, systemd timer, etc ...) rather than manage the execution itself which results in longer agent run times and increases the likelihood of overlapping agent runs.

[nickanderson]

Oh, and if your really curious how much time a host would splay itself, you could check out this old python script, cfe-rsplaytime which takes hostname, IP and UID and calculates the splay using the same methodology as core.

https://loicpefferkorn.net/cfe-rsplaytime/