Tco - A Multi-Purpose Tool for Tocco Developers

I’ve been working on on streamlining my script collection. To this end, I rewrote much of the code and put it into a new repository. Until now, with my old collection of scripts, a lot of code was duplicated, non-portable and depended on local, manual setup.

With the new code and repository, this deficiencies no longer exist. Shared functionality, like connecting to a remote Postgres server, working with the Ansible repository or extracting settings from Ansible is now implemented in an easily reusable way.

Moreover, things like manually cloning an ansible or nice2 repositories in a predefined places, or manually updating them is no longer required. This is all done in the background. As added bonus, I also added tests and documentation for shared code, and I’m making the code public to anyone interested in it. Some functionality is probably only of interest to me, or DevOps at best, other functionality may very well proof useful for others.

Installation

Tco still still rather experimental and as an early adopter, you’ll have to build it yourself:

  1. Install package manager / compiler

    Instructions at https://rustup.rs/

  2. Build and install:

    cargo install --git https://gitlab.com/toccoag/tco.git

Similarly, updates need to be applied manually too:

  1. Update compiler and toolchain:

    rustup update

  2. Update tco:

    cargo install --git https://gitlab.com/toccoag/tco.git

Note:

Many tco subcommands expect Ansible to be set up according to our documentation and that you have a ssh key registered with GitLab.

Implemented Functionality

Here some examples to show the kind of functionality available with tco.

--help

As with the majority of my CLI tools, use --help for more information:

$ tco --help
Usage: tco [OPTIONS] <COMMAND>

Commands:
  customer-undeployed
          Customer-specific commits not yet deployed. (cu)
  undeployed
          Commits not yet deployed. (u)
  build-dates
          List build dates for all installations (bd)

--- cut ---

  status
          Show installation status
  help
          Print this message or the help of the given subcommand(s)

Options:
  -u, --update-repos-secs <UPDATE_REPOS_SECS>
          Update interval for locally cached Git repositories.

          Update repositories that have not been updated for given seconds before use. Use -1 to disable updates completely.

          [default: 900]

--- cut ---

Of course, this works for subcommands, like build-dates too:

$ tco build-dates --help
List build dates for all installations (bd)

All changes up to (approximately) the build date have been deployed. Note that after a prod deployment test and prod will have the same build date. This because the application / Docker image is copied and not rebuilt.

Installations are sorted by build dates, least recently deployed first.

Short: "bd"

Usage: tco build-dates [OPTIONS]

Options:
  -e, --environment <ENVIRONMENT>
          Limit output to selected environment

          [default: all]
          [possible values: all, prod, test]

  -f, --format <FORMAT>
          Format to use for build date / time

          [default: relative]
          [possible values: date, date-time, relative]

      --parallel <PARALLEL>
          Max. number of parallel HTTP requests

          [default: 50]

  -h, --help
          Print help (see a summary with '-h')

tco *undeployed - Changes Pending Deployment

Various *undeployed subcommands exist helping you to identify changes not yet deployed. From the --help output:

  customer-undeployed
          Customer-specific commits not yet deployed. (cu)
  undeployed
          Commits not yet deployed. (u)
  customer-prod-undeployed
          Customer-specific commits not yet deployed to prod but already on test. (cpu)
  prod-undeployed
          Commits not yet deployed to prod but already on prod. (pu)

For instance, you can show all customer-specific changes not yet deployed:

$ tco customer-undeployed example
commit ba3b03b378bb21041e297fa4bc1e636d34ff1434
Author: Jane Doe <jdoe@tocco.ch>
Date:   Wed Feb 8 16:31:58 2023 +0100

    check in manual changes from prod system

    Closes: BS-8384

In this case, we see that there is one customer-specific change in the nice2 repository, not yet deployed to example. More specifically there is one commit touching a file in customer/example/ within the nice2 repository which has not yet been deployed.

I find this group of subcommands useful in order to see what bugfixes are missing on an installation or if the installation can be deployed safely (without risking rolling out a customer-specific change to production).

Alternately, you can use the short version (cu) of the command (shown in parenthesis in the help output):

$ tco cu example

tco db-connect - PSQL Console

You can connect to a DB directly, without knowing its location or name.

Connect to the DB of installation example:

$ tco db-connect example
psql (15.2 (Ubuntu 15.2-1.pgdg20.04+1))
Type "help" for help.

nice_example=#

Or, alternatively, the history DB:

$ tco db-connect -t history example
psql (15.2 (Ubuntu 15.2-1.pgdg20.04+1))
Type "help" for help.

nice_example_history=#

tco db-ssh - SSH connection to DB server

You can also connect to the DB server of installation example via SSH:

$ tco db-ssh example

Or execute a specific command on the server:

$ tco db-ssh example -- ls -lh dumps/

As with any CLI tool using the GNU-syntax for , use -- to ensure -lh is parsed as option to tco.

tco sessions - Logged in Users

Show all users currently logged in on installation example:

$ tco sessions example
IP               SESSION REFRESH  NAME                           USER NAME
123.45.123.45     1 min 50 s ago  Jane Doe                       jdoe@tocco.ch
Tocco Office      6 min 30 s ago  John Doe                       jdoe2@tocco.ch
210.210.98.98     2 min 31 s ago  Event Exporter                 export_api

I find this useful to find out, on test system in particular, if it is a good time to apply and test a configuration change or deploy a bug fix. With no user, there is little chance of causing a disruption for someone else.

tco status - Status of a Running Installation

/status-tocco page with some bonus details:

$ tco status master
Status of "master"

Build:
    Version:  3.6
    Revision: 23d9f292d02
    Build time: 2023-03-10 06:04:12 +01:00 (0 days, 4 hours ago)

Environment:
    Host / pod:  nice-604-sxdtj
    Environment: test

Java Heap Memory:
    Max. memory:        1532 MiB
    Total memory:       1388 MiB (91%)
    Free memory:         155 MiB
    Used memory:        1233 MiB (80%)
    Total free memory:   299 MiB (20%)

    Percentages are relative to "max. memory".

This is mostly just an easy way to access /status-tocco. However, the following additional information is added:

  • A relative build time.
  • Where the build time isn’t available, the commit time is looked up and shown instead.
  • Percentages are added to memory stats.

tco info - Ansible settings

Show settings from Ansible:

$ tco info master
Installation "master" (v1000.0.0)

Main DB:
    name:   nice_master
    server: db3.stage.tocco.cust.vshn.net

History DB:
    name:   nice_master_history
    server: db3.stage.tocco.cust.vshn.net

With some more details:

$ tco info master -v
Installation "master" (v1000.0.0)

Main DB:
    name:   nice_master
    server: db3.stage.tocco.cust.vshn.net
    size:   0.7 GiB

History DB:
    name:   nice_master_history
    server: db3.stage.tocco.cust.vshn.net
    size:   0.2 GiB

This is currently very minimal but I’ll likely add some more information in the future like mail and route settings.

tco build-dates

Show build/commit dates for all installations:

$ tco build-dates -f date
--- cut ---
  6 d 23 h  * abctest
  6 d 23 h    example
  4 d  7 h    exampletest
--- cut ---

* -> Build time unavailable, time of Git commit shown instead.

Use -f date or -f date-time to change the time format.

I hope this will help me to roll out bugfixes more rapidly. Showing where customer-specific commits are pending and, thus, a deployment isn’t possible is already planned.

tco dbs unused - Finding Unused DBs

All unused DBs on any DB server:

$ tco dbs unused
SIZE       LAST LOGIN    SERVER      DB
         / NEWEST ENTRY
 2.1 GiB  13 d 21 h      db3.stage   nice_abctest
 0.4 GiB   0 d 23 h      db3         nice_example_backup
 0.0 GiB                 db5.prod    bitwarden
 0.0 GiB                 db1.stage   nice_abctest_history
 0.0 GiB                 db1.stage   nice_abc_history

This is mostly useful to free space on the servers. A DB is shown as unused when no connection to it exists. To verify a DB is indeed unused, LAST LOGIN is added for Nice DBs, and NEWEST ENTRY for history DBs.

See tco dbs unused --help for some useful.

tco dbs failing-tasks - Show Repeatedly Failing Tasks

Of all installations, show tasks which failed the last three runs:

$ tco dbs failing-tasks
LAST FAILURE         VERSION  INSTALLATION          TASK
--- cut ---
  9 d  8 h           v3.5     tocco                 CalculateReleasePlanArticlePriceBatchJob
  0 d  7 h           v3.5     tocco                 CleanupModuleMeasurementBatchJob
  9 d  8 h           v3.5     toccotest             CalculateReleasePlanArticlePriceBatchJob
  0 d  7 h           v3.5     toccotest             CleanupModuleMeasurementBatchJob
  9 d  8 h           v3.4     toccotestold          CalculateReleasePlanArticlePriceBatchJob
  0 d  7 h           v3.4     toccotestold          CleanupModuleMeasurementBatchJob
--- cut ---

See tco dbs failing-tasks --help for options.

This output is based on table nice_task_execution and is mostly useful to see failing batch jobs.

Where Next

There is various features I’d like to add:

  • OpenShift integration: viewing and filtering logs
  • copying DBs with no need to know where a DB is located
  • SPF / DKIM validation which allows running a check across all domains (in a CI)

Let me know if you find tco useful and you’d like to have certain features added or add them yourself.

I’m also looking into how to ship this and other scripts and configurations. I currently believe the best approach would be to ship Debian packages (which are used Ubuntu too). Ideally we could provide a tocco-dev-full meta-package that installs everything, including Ansible, Java, etc.