cfbs add a local git repo?

[t3mcvay]

I'm starting to learn how to migrate existing CFengine policy to a cfbs managed set. Our policy repo is a local git repo on disk, and we have a use case/need to NOT make it accessible via something like github. Is it possible to have a local git repo on disk and then have 'cfbs add /path/to/local/policy' actually be able to recognize that /path/to/local/policy is actually a local repo?

[olehermanse]

@t3mcvay Hello, thanks for opening a discussion and good question;

I asssume your policy (on disk) is a fork of the MPF?

We are working on some features to make this transition easier, but one thing to consider is that we want cfbs to produce reproducible builds. In your case, this means that we don't want it to just blindly use what is in ~/some/path/to/masterfiles/, but we want to it have some checksums / version information in addition to just the path. Currently, without waiting for us to add new features, there's a couple of ways to achieve this;

Git submodules

A submodule is one mechanism to achieve this. The "normal" way would be to put your dependency (the classic policy set) somewhere on a git server, can be GitHub private repo, on-prem instance of GitLab, or something else. You can then add that URL as a git submodule:

$ git submodule add https://server/some/repo/url

Since you mentioned not using something like GitHub, instead using local files, it is also possible to use local file paths, but due to security aspects, you will have to explicitly enable submodules via local filesystem paths;

$ git config --global protocol.file.allow always
$ git submodule add --force ../masterfiles

Warning: I wouldn't generally recommend doing this - especially if it's your main work machine where you might git clone some projects and then run a Dockerfile from that project for example. That opens you up to the project referencing some local files via paths and then those being transferred to the docker container running potentially untrusted code. Other similar scenarios exist.

For how to work with git submodules, I'd use the official documentation:

https://git-scm.com/book/en/v2/Git-Tools-Submodules

After you've added the submodule, the last part would be adding the module / editing the cfbs.json.
See the section at the end for how your CFEngine Build project (cfbs.json) should look like.

Copying in the files explicitly

The other option would be to copy in the files, with something like rsync;

$ cfbs init --non-interactive --masterfiles=no
$ rsync -r --exclude '.git*' --exclude '*.md' --exclude '*.org' --exclude 'tests/' ~/some/path/to/masterfiles/ ./masterfiles/ --delete
$ cfbs --non-interactive add ./masterfiles/

(Edit the rsync command with the right path and feel free to exclude more things).

Updating

With this rsync setup, this is how you'd update from the local path:

$ rsync -r --exclude '.git*' --exclude '*.md' --exclude '*.org' --exclude 'tests/' ~/some/path/to/masterfiles/ ./masterfiles/ --delete
$ git add ./masterfiles/
$ git commit -m "Updated ./masterfiles from ~/some/path/to/masterfiles/"

For convenience, you can add these commands to an alias, or shell script, if you like.

CFEngine Build project / Build steps

Whether you're using rsync or git submodules, you can edit the cfbs.json to do what's needed. For example:

{
  "name": "Example project",
  "description": "Example description",
  "type": "policy-set",
  "git": true,
  "build": [
    {
      "name": "./masterfiles/",
      "description": "Our old masterfiles folder added as a module, synced using rsync",
      "tags": ["local"],
      "added_by": "cfbs add",
      "steps": ["copy ./ ./"]
    }
  ]
}

Or, if your masterfiles repo requires running some script (like prepare.sh):

{
  "name": "Example project",
  "description": "Example description",
  "type": "policy-set",
  "git": true,
  "build": [
    {
      "name": "./masterfiles/",
      "description": "Our old masterfiles folder added as a module, synced using rsync",
      "tags": ["local"],
      "added_by": "cfbs add",
      "steps": ["run ./prepare.sh -y", "copy ./ ./"]
    }
  ]
}

Note: Ensure the rsync command is copying what is needed for your script to run.

What's next?

We're adding new commands and features to make this nicer for you. Hopefully the workarounds above can work, and we'll release new commands in cfbs soon which will make this easier for you.

When you have a working project, you can use the cfbs tooling and start adding modules from build.cfengine.com. We also recommend you start working towards splitting your classic masterfiles into 2, one part which you maintain (your custom policy files and modifications), and one part which we maintain (the original, versioned, MPF).

[t3mcvay]

Hello @olehermanse

Thanks for the quick reply. No, your initial assumption is incorrect. We have a long history with CFengine (way back to an early 2.X version). We've been carrying that initial custom policy along to now, and do not have any version of the MPF in our CM repo. That 'old' policy works OK with CFengine 3 (currently working with 3.10).

But, we think it's time to at least see if incorporating some version of the MPF is feasible at this point.

The current thought is to migrate (1-by-1) our existing policy into the equivalent custom policy (as I understand right now, which would fit under ./services/). Also -- if something like the submodules example you so kindly provided could be used we could not have a URL as many of our deployed systems are airgapped and we literally just use policy stored in a local repo to administer and setup hosts on a network. Also I should mention that we have developed (over the years) tools to assist in administering this (such as dynamic generation of things like sudoers, and other custom configurations per host depending upon their function), and that needs to continue as that allows us to write fairly generic policy which will work across a variety of network and security postures.

[nickanderson]

Interesting stuff. I wonder if cfbs would help you with managing those various air-gapped policy sets.

How much of that policy can't leave the air-gapped network? Or maybe a better question is much of the policy is specific to individual air gapped networks?

Would it be easier to use cfbs to build a policy set on your normal network and then sneaker-net the built result over to the air gapped network

[t3mcvay]

Hi @nickanderson

See - that's the $64 question we're trying to answer. We do fine now with our custom policy set and (most generally) use the same policy across all of our networks as much as possible. We can do this as we dynamically extract data (IPs, hostnames, etc...) from a dataset which is unique per network. So the policy itself is pretty generic, but the 'dereferenced variables' may not be, if that makes sense. Given how we deploy to our airgapped newtworks the answer is: that is also what we're trying to figure out. I think there may be some appetite for using cfengine vetted modules from github, but not just random things found across the interwebs.

Cheers.

[nickanderson]

I think I understand.

Do you have any policies that cant exist off gapped network?

Do you have any dynamic inputs (e.g. loading a policy file based on some variable)?

Do you have have any unique input file names per gapped network (this policy file is only used on gapped network a)?