Announcing orchestrator-agent
This post was written by Shlomi Noach
orchestrator-agent is a side-kick, complementary project of orchestrator, implementing a daemon service on one’s MySQL hosts which communicates with and accepts commands from orchestrator, built with the original purpose of providing an automated solution for provisioning new or corrupted slaves.
It was built by Outbrain, with Outbrain’s specific use case in mind. While we release it as open source, only a small part of its functionality will appeal to the public (this is why it’s not strictly part of the orchestrator project, which is a general purpose, wide-audience solution). Nevertheless, it is a simple implementation of a daemon, such that can be easily extended by the community. The project is open for pull-requests!
A quick breakdown of orchestrator-agent is as follows:
- Executes as a daemon on linux hosts
- Interacts and invokes OS commands (via bash)
- Does not directly interact with a MySQL server running on that host (does not connect via mysql credentials)
- Expects a single MySQL service on host
- Can control the MySQL service (e.g. stop, start)
- Is familiar with LVM layer on host
- Can take LVM snapshots, mount snapshots, remove snapshots
- Is familiar with the MySQL data directory, disk usage, file system
- Can send snapshot data from a mounted snapshot on a running MySQL host
- Can prepare data directory and receive snapshot data from another host
- Recognizes local/remote datacenters
- Controlled by orchestrator, two orchestrator-agents implement an automated and audited solution for seeding a new/corrupted MySQL host based on a running server.
Offline, hard-copy backups aside, Outbrain implements quick backups via LVM snapshots. Some slaves (depending on strength, data center etc.) take the role of snapshot servers. These slaves take an LVM snapshot once per day, and keep it open unmounted. At any given time we might have multiple open snapshots on a snapshot server. At a few minutes notice, we are able to restore MySQL to yesterday’s, the day before, two days before, … state. This only takes mounting of the desired snapshot and starting MySQL over the mounted volume.
But we also use these snapshots on a daily basis, putting them constantly to practice (hence validating this backup method routinely): we provision new MySQL servers based on those snapshots. A new or corrupted server can be seeded with a snapshot data (preferably from a host in same datacenter). We don’t need to perform a backup now. We can take the one from tonight.
Orchestrator-agent was invented with this particular use case in mind, and orchestrator was extended to support these operations. Orchestrator can act as the controller of a seed/provisioning operation, getting info from/to orchestrator-agent and coordinating two agents to send/receive the data.
Should you use orchestrator-agent?
If, like us, you use LVM snapshots, at least as a partial form of backup – bingo!
Otherwise, orchestrator-agent can assist in stopping/starting the MySQL service, tailing the error log; simple stuff like that — probably too much fuss for such basic operations.
But it can be easily extended to support other forms of backup (though do note that orchestrator-agent does not and should not perform scheduled backups, merely manipulate them); so if you like to contribute code into orchestrator-agent, please let us know! It would be nice to have a solution that works for multiple use cases.
Get it
orchestrator-agent is released under the Apache 2.0 license, and is free to use.
Pre-compiled, RPM and Deb packages can be found on the releases page.