Filesystem Cluster Synchronization with Small Footprint

Syncing a filesystem in a cluster is one of the most challenging things. There are many ways to do it:

• iSCSI with GFS2: you require local computers with a fast iSCSI connection to make it work;
• Glusterfs: it is slow if your nodes are far from each other (as it is a synchronous filesystem);
• NFS: famous because it is one of the slowest filesystems;
• Syncthing: big footprint, especially when scanning the filesystem, file synchronization could be delayed.

These are some of the many approaches you may take. I am going to talk about a new one.

• CSync2 with Lsyncd: small footprint with almost real-time synchronization.

Installation

If you are using my RPM repo (for RHEL Linux distributions), just type: yum -y  install csync2 lsyncd xinetd sqlite-devel. This should be more than enough.

I took special effort forward porting xinetd to RHEL9.

Configuration

Make sure the nodes are resolvable to each other. The easiest way to do it is by adding an entry into the /etc/hosts file and syncing the file between your nodes.

Later, check if the directory /var/csync2/backup exists, if it doesn't, please create it. Type: mkdir -p /var/csync2/backup.

After that, type csync2 -k /etc/csync2.key. This will create a unique key, and copy this key in all your nodes (filename and directory are important).

Here comes the interesting part, create a file named /etc/csync2_key.cfg. Change the key to anything you want and put the following content:

nossl * *;
group key
{
 host node1 node2 node3;
 key /etc/csync2.key;
 include /var/lib/freeswitch/recordings;
 include /var/lib/freeswitch/storage;
 auto younger;
 backup-directory /var/csync2/backup;
 backup-generations 3;
}

Copy this file to all other nodes. The next step is to build the csync2 database in each node; to do that, type: csync2 -xXrvlB -C key. If all is okay, no errors should be displayed.

Csync2 comes as a xinetd service. Start xinetd and make sure it starts each time the nodes are restarted, to do that type: systemctl enable xinetd, and systemctl start xinetd. Csync2 works using port 30865/TCP, make sure you have the proper firewall rules.

Lsyncd configuration is simpler. Edit the file /etc/lsyncd.conf and write the following content:

settings {
  logident = "lsyncd",
  logfacility = "user",
  logfile = "/var/log/lsyncd/lsyncd.log",
  statusFile = "/var/log/lsyncd/status.log",
  statusInterval = 1
}
initSync = {
  delay = 1,
  maxProcesses = 1,
  action = function(inlet)
    local config = inlet.getConfig()
    local elist = inlet.getEvents(function(event)
      return event.etype ~= "Init"
    end)
    local directory = string.sub(config.source, 1, -2)
    local paths = elist.getPaths(function(etype, path)
      return "\t" .. config.syncid .. ":" .. directory .. path
    end)
    log("Normal", "Processing syncing list:\n", table.concat(paths, "\n"))
    spawn(elist, "/usr/sbin/csync2", "-C", config.syncid, "-x")
 end,
 collect = function(agent, exitcode)
  local config = agent.config
  if not agent.isList and agent.etype == "Init" then
    if exitcode == 0 then
      log("Normal", "Startup of '", config.syncid, "' instance finished.")
    elseif config.exitcodes and config.exitcodes[exitcode] == "again" then
      log("Normal", "Retrying startup of '", config.syncid, "' instance.")
      return "again"
    else
      log("Error", "Failure on startup of '", config.syncid, "' instance.")
      terminate(-1)
    end
    return
  end
  local rc = config.exitcodes and config.exitcodes[exitcode]
  if rc == "die" then
    return rc
  end
  if agent.isList then
    if rc == "again" then
      log("Normal", "Retrying events list on exitcode = ", exitcode)
    else
      log("Normal", "Finished events list = ", exitcode)
    end
  else
    if rc == "again" then
      log("Normal", "Retrying ", agent.etype, " on ", agent.sourcePath, " = ", exitcode)
    else
      log("Normal", "Finished ", agent.etype, " on ", agent.sourcePath, " = ", exitcode)
    end
  end
  return rc
 end,
 init = function(event)
  local inlet = event.inlet;
  local config = inlet.getConfig();
  log("Normal", "Recursive startup sync: ", config.syncid, ":", config.source)
  spawn(event, "/usr/sbin/csync2", "-C", config.syncid, "-x")
 end,
 prepare = function(config)
  if not config.syncid then
    error("Missing 'syncid' parameter.", 4)
  end
  local c = "csync2_" .. config.syncid .. ".cfg"
  local f, err = io.open("/etc/" .. c, "r")
  if not f then
    error("Invalid 'syncid' parameter: " .. err, 4)
  end
  f:close()
end
}
local sources = {
 ["/var/lib/freeswitch/recordings"] = "key",
 ["/var/lib/freeswitch/storage"] = "key"
}
for key, value in pairs(sources) do
 sync {initSync, source=key, syncid=value}
end

 Pay attention to the last lines. The paths and the key must match with the csync2 configuration. Copy this file to all the nodes.

DISCLAIMER: I am not the author of the lsyncd.conf file.

When you are done, just type systemctl enable lsyncd, and systemctl start lsyncd.

Good luck!