How to Secure a Previously Insecure Cluster

Begin with an insecure cluster.

cockroach start --insecure --store=node1 --listen-addr=localhost:26257 --http-addr=l 
ocalhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* WARNING: RUNNING IN INSECURE MODE!
* 
* - Your cluster is open for any client that can access localhost.
* - Any user, even root, can log in without providing a password.
* - Any user, connecting as root, can read or write any data in your cluster.
* - There is no network encryption nor authentication, and thus no confidentiality.
* 
* Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html
*
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress. 
*
bash-3.2$ cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=l 
ocalhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* WARNING: RUNNING IN INSECURE MODE!
* 
* - Your cluster is open for any client that can access localhost.
* - Any user, even root, can log in without providing a password.
* - Any user, connecting as root, can read or write any data in your cluster.
* - There is no network encryption nor authentication, and thus no confidentiality.
* 
* Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html
*
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress. 
bash-3.2$ cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=l 
ocalhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* WARNING: RUNNING IN INSECURE MODE!
* 
* - Your cluster is open for any client that can access localhost.
* - Any user, even root, can log in without providing a password.
* - Any user, connecting as root, can read or write any data in your cluster.
* - There is no network encryption nor authentication, and thus no confidentiality.
* 
* Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html
*
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress. 
bash-3.2$ cockroach init --insecure --host=localhost:26257
Cluster successfully initialized
bash-3.2$ grep 'node starting' node1/logs/cockroach.log -A 11
CockroachDB node starting at 2020-04-27 13:39:56.808756 +0000 UTC (took 19.4s)
build:               CCL v20.1.0-rc.1 @ 2020/04/10 01:53:13 (go1.13.9)
webui:               http://localhost:8080
sql:                 postgresql://root@localhost:26257?sslmode=disable
RPC client flags:    cockroach <client cmd> --host=localhost:26257 --insecure
logs:                /Users/artem/Downloads/secure/node1/logs
temp dir:            /Users/artem/Downloads/secure/node1/cockroach-temp775384998
external I/O path:   /Users/artem/Downloads/secure/node1/extern
store[0]:            path=/Users/artem/Downloads/secure/node1
storage engine:      rocksdb
status:              initialized new cluster
clusterID:           42062b66-c4c5-4167-bae6-9b0e2ebc47f5
bash-3.2$ cockroach sql --insecure --host=localhost:26257
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter ? for a brief introduction.
#
root@localhost:26257/defaultdb> 
CREATE DATABASE bank;
CREATE DATABASE

Time: 4.644ms

root@localhost:26257/defaultdb> 
CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMA 
L);
CREATE TABLE

Time: 5.005ms

root@localhost:26257/defaultdb> 
INSERT INTO bank.accounts VALUES (1, 1000.50);
INSERT 1

Time: 5.354ms

root@localhost:26257/defaultdb> 
SELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 1.465ms

root@localhost:26257/defaultdb> q
bash-3.2$ # connect from another node
bash-3.2$ cockroach sql --insecure --host=localhost:26258
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter ? for a brief introduction.
#
root@localhost:26258/defaultdb> 
SELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 7.537ms

root@localhost:26258/defaultdb> q

Secure cluster take over:

bash-3.2$ mkdir certs my-safe-directory
bash-3.2$ cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key
bash-3.2$ cockroach cert create-node localhost $(hostname) --certs-dir=certs --ca-key=my-safe- 
directory/ca.key
bash-3.2$ cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.ke 
y

Stop each node one by one and restart in secure mode, because each node has the same hostname, we will reuse the cert:

bash-3.2$ ps aux | grep cokroach
artem            73363   1.9  0.7  4843760 220372 s003  S     9:39AM   0:05.50 cockroach start --insecure --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
artem            73368   1.0  0.5  4787908 175784 s003  S     9:39AM   0:04.55 cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
artem            73371   0.9  0.5  4787176 170988 s003  S     9:39AM   0:04.20 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
artem            73461   0.0  0.0  4417788    840 s003  S+    9:42AM   0:00.00 grep cockroach
bash-3.2$ kill 73363 # stopping node1
bash-3.2$ cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http 
-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background
bash-3.2$ clear
bash-3.2$ ps aux | grep cockroach
artem            73371   4.3  0.5  4795368 174144 s003  S     9:39AM   0:04.86 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
artem            73368   3.6  0.5  4796100 177476 s003  S     9:39AM   0:05.18 cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
artem            73470   0.9  0.2  4622236  60644 s003  S     9:42AM   0:00.39 cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
artem            73476   0.0  0.0  4399356    800 s003  S+    9:42AM   0:00.00 grep cockroach
kill 73368 # node 2
bash-3.2$ cockroach start --certs-dir=certs --store=node2 --listen-addr=localhost:26258 --http 
-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 --background
bash-3.2$ clear
bash-3.2$ ps aux | grep cockroach
artem            73470   2.1  0.3  4700764  86980 s003  S     9:42AM   0:01.51 cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
artem            73486   1.8  0.2  4694508  72112 s003  S     9:43AM   0:00.59 cockroach start --certs-dir=certs --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
artem            73371   0.7  0.5  4797672 178068 s003  S     9:39AM   0:05.85 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
artem            73492   0.0  0.0  4410620    848 s003  S+    9:43AM   0:00.00 grep cockroach
bash-3.2$ kill 73371 # node 3
bash-3.2$ cockroach start --certs-dir=certs --store=node3 --listen-addr=localhost:26259 --http 
-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 --background
bash-3.2$ grep 'node starting' node1/logs/cockroach.log -A 11
CockroachDB node starting at 2020-04-27 13:43:27.451629 +0000 UTC (took 44.3s)
build:               CCL v20.1.0-rc.1 @ 2020/04/10 01:53:13 (go1.13.9)
webui:               https://localhost:8080
sql:                 postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt
RPC client flags:    cockroach <client cmd> --host=localhost:26257 --certs-dir=certs
logs:                /Users/artem/Downloads/secure/node1/logs
temp dir:            /Users/artem/Downloads/secure/node1/cockroach-temp837519533
external I/O path:   /Users/artem/Downloads/secure/node1/extern
store[0]:            path=/Users/artem/Downloads/secure/node1
storage engine:      rocksdb
status:              restarted pre-existing node
clusterID:           42062b66-c4c5-4167-bae6-9b0e2ebc47f5
bash-3.2$ # connect to the cluster with client certs bash-3.2$ cockroach sql --certs-dir=certs --host=localhost:26257 # # Welcome to the CockroachDB SQL shell.  # All statements must be terminated by a semicolon.  # To exit, type: q.  # # Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client) # Cluster ID: 42062b66-c4c5 -4167-bae6-9b0e2ebc47f5 # # Enter ?  for a brief introduction.  # root@localhost:26257/defaultdb> show databases;  database_name ----------------- bank defaultdb postgres system (4 rows) Time: 2.002ms root@localhost:26257/defaultdb> use bank;  SET Time: 993µs root@localhost:26257/bank> use bank;  show databases;  SELECT * FROM bank.accounts;  ‎[28Gq�[K
�[28GSELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 5.893ms

root@localhost:26257/bank> q
bash-3.2$ # connect to another node 
bash-3.2$ # cockroach sql --certs-dir=certs --host=localhost:26259
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter ? for a brief introduction.
#
root@localhost:26259/defaultdb> 
SELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 5.105ms

root@localhost:26259/defaultdb> q

Once complete, you may also validate the certs by creating a user with admin privileges and navigating to the certificates page in the Admin UI, such as https://<address of node with new certs>:8080/#/reports/certificates/local and view all of the available certificates.

CREATE USER roach WITH PASSWORD 'cockroach';

GRANT ADMIN TO roach;

SHOW ROLES;

    username    |  options   | member_of
----------------+------------+------------
  admin         | CREATEROLE | {}
  roach         |            | {admin}
  root          | CREATEROLE | {admin}
(3 rows)

Certificate Authority node certificate client certificate

.

Leave a Comment