Block Busting

GHE server version: 2.19.2

Reference

https://github.community/t5/GitHub-Enterprise-Best-Practices/High-Availability-and-Disaster-Recovery-for-GitHub-Enterprise/ba-p/11725

https://help.github.com/en/enterprise/2.19/admin/installation/configuring-github-enterprise-server-for-high-availability

HA (Normal Case - Primary & Replica normal status)

Primary Side

Mode change to 'Maintnance mode'

Run command

• ghe-maintenance -s

Replica Side

Change AWS Route 53 IP address

• primary → replica

Run command (GHE status change replica to primary)

• ghe-repl-promote

admin@ip-172-3?-???-???:~$ ghe-repl-promote
Warning: You are about to promote this Replica node
Promoting this Replica will tear down replication and enable maintenance mode on the current Primary.
All other Replicas need to be re-setup to use this new Primary server.
 
Proceed with promoting this appliance to Primary? [y/N] y
Enabling maintenance mode on the primary to prevent writes ...
Stopping replication ...
  | Stopping Pages replication ...
  | Stopping Git replication ...
  | Stopping Alambic replication ...
  | Stopping git-hooks replication ...
  | Stopping MySQL replication ...
  | Stopping Redis replication ...
  | Stopping Consul replication ...
  | Success: Replication was stopped for all services.
  | To disable replica mode and remove all replica configuration, run 'ghe-repl-teardown'.
Switching out of replica mode ...
  | Dec 04 07:41:45 Preparing storage device...
  | Dec 04 07:41:46 Updating configuration...
  | Dec 04 07:41:46 Reloading system services...
  | Dec 04 07:42:18 Running migrations...
  | Dec 04 07:42:54 Reloading application services...
  | Dec 04 07:43:18 Done!
  | jq: error (at :0): Cannot index number with string "settings"
  | Success: Replication configuration has been removed.
  | Run `ghe-repl-setup' to re-enable replica mode.
Applying configuration and starting services ...
  | ERROR: cannot launch /usr/local/bin/ghe-single-config-apply - run is locked
admin@ip-172-3?-???-???:~$

Former Primary Side

Former primary mode change to replica

Run command

• ghe-repl-setup 172.3?.???.???
• ghe-repl-start
• ghe-repl-status
• ghe-maintenance -u (maintnace mode unset)

admin@githubtest-testdomain.io:~$ ghe-repl-setup 172.3?.???.???
Warning: This appliance is or has been a configured appliance.
Proceeding will overwrite data on this appliance.
 
Proceed with initializing this appliance as a replica? [y/N] y
Verifying ssh connectivity with 172.3?.???.??? ...
Connection check succeeded.
Updating Elasticsearch configuration ...
Copying license and settings from primary appliance ...
 --> Importing SSH host keys...
 --> The SSH host keys on this appliance have been replaced to match the primary.
 --> Please run 'ssh-keygen -R 172.3?.XXX.XXX; ssh-keygen -R "[172.3?.XXX.XXX]:122"' on your client to prevent future ssh warnings.
Copying custom CA certificates from primary appliance ...
Success: Replica mode is configured against 172.3?.???.???.
To disable replica mode and undo these changes, run 'ghe-repl-teardown'.
Run 'ghe-repl-start' to start replicating from the newly configured primary.
 
admin@githubtest-testdomain.io:~$ ghe-repl-start
Verifying ssh connectivity with 172.3?.???.??? ...
Updating configuration...
Validating configuration
Updating configuration for githubtest-testdomain.io-primary (172.3?.???.???)
Configuration Updated
Configuration Phase 1
githubtest-testdomain.io-replica: Dec 04 07:49:43 Preparing storage device...
githubtest-testdomain.io-replica: Dec 04 07:49:45 Updating configuration...
githubtest-testdomain.io-replica: Dec 04 07:49:45 Reloading system services...
githubtest-testdomain.io-replica: Dec 04 07:50:04 Done!
githubtest-testdomain.io-primary: Dec 04 07:49:43 Preparing storage device...
githubtest-testdomain.io-primary: Dec 04 07:49:45 Updating configuration...
githubtest-testdomain.io-primary: Dec 04 07:49:45 Reloading system services...
githubtest-testdomain.io-primary: Dec 04 07:50:10 Done!
Configuration Phase 2
githubtest-testdomain.io-replica: Dec 04 07:50:13 Running migrations...
githubtest-testdomain.io-replica: Dec 04 07:50:13 Done!
githubtest-testdomain.io-primary: Dec 04 07:50:15 Running migrations...
githubtest-testdomain.io-primary: Dec 04 07:50:47 Done!
Configuration Phase 3
githubtest-testdomain.io-primary: Waiting for services to be active...
githubtest-testdomain.io-primary: Dec 04 07:51:06 Reloading application services...
githubtest-testdomain.io-primary: Dec 04 07:51:29 Done!
githubtest-testdomain.io-replica: Dec 04 07:50:48 Reloading application services...
githubtest-testdomain.io-replica: Dec 04 07:51:59 Done!
Finished cluster configuration
Success: replication is running for all services.
Run `ghe-repl-status' to monitor replication health and progress.
 
 
 
admin@githubtest-testdomain.io:~$ ghe-repl-status
OK: mysql replication is in sync
OK: redis replication is in sync
OK: elasticsearch cluster is in sync
OK: git replication is in sync
OK: pages replication is in sync
OK: alambic replication is in sync
OK: git-hooks replication is in sync
OK: consul replication is in sync
 
 
 
admin@githubtest-testdomain.io:~$ ghe-maintenance -u

HA (Disaster case - Primary EC2 terminated)

If Primary EC2 instance terminated.

Just run this command replica side

• ghe-repl-promote

And change Route53 DNS IP address to replica

admin@githubtest-testdomain.io-replica:~$ ghe-repl-promote
Warning: You are about to promote this Replica node
Promoting this Replica will tear down replication and enable maintenance mode on the current Primary.
All other Replicas need to be re-setup to use this new Primary server.
 
Proceed with promoting this appliance to Primary? [y/N] y
ssh: connect to host 172.3?.???.??? port 122: Connection timed out
Warning: Primary node is unavailable.
Warning: Performing hard failover without cleaning up on the primary side.
Stopping replication ...
  | Skipping Pages, Alambic, git-hooks and Git replication cleanup on primary ...
  | Stopping MySQL replication ...
  | Stopping Redis replication ...
  | Stopping Consul replication ...
  | Success: Replication was stopped for all services.
  | To disable replica mode and remove all replica configuration, run 'ghe-repl-teardown'.
Switching out of replica mode ...
  | ssh: connect to host 172.3?.???.??? port 122: Connection timed out
  | ssh: connect to host 172.3?.???.??? port 122: Connection timed out
  | ssh: connect to host 172.3?.???.??? port 122: Connection timed out
  | ssh: connect to host 172.3?.???.??? port 122: Connection timed out
  | ssh: connect to host 172.3?.???.??? port 122: No route to host
  | jq: error (at :0): Cannot index number with string "settings"
  | jq: error (at :0): Cannot index number with string "settings"
  | Success: Replication configuration has been removed.
  | Run `ghe-repl-setup' to re-enable replica mode.
Applying configuration and starting services ...
Success: Replica has been promoted to primary and is now accepting requests.

GHE server version: 2.19.2

Reference

https://github.community/t5/GitHub-Enterprise-Best-Practices/High-Availability-and-Disaster-Recovery-for-GitHub-Enterprise/ba-p/11725

https://help.github.com/en/enterprise/2.19/admin/installation/configuring-github-enterprise-server-for-high-availability

Create 1 more EC2 instance for GHE image.

• https://sfixer.tistory.com/entry/Github-enterprise-server-setup-1-setup

Replication server could use same license file.

Select installation type

• Configure as Replica

After finish setup.

Run command (Replica VM)

• ghe-repl-setup $PRIMARY_VM_IP_ADDRESS

admin@ip-172-3?-???-???:~$ ghe-repl-setup 172.3?.???.???
Generating public/private ed25519 key pair.
/home/admin/.ssh/id_ed25519 already exists.
Overwrite (y/n)? Your identification has been saved in /home/admin/.ssh/id_ed25519.
Your public key has been saved in /home/admin/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:t3C46zQrYB/???????????????????????s1dc/e4Ho admin-ssh-key
The key's randomart image is:
+--[ED25519 256]--+
| o++o. . .       |
|..o.* + . o      |
|.. = O     +     |
|  . * o  .o o    |
|   o * oS.oo .   |
|    B O o=..     |
|   o * *+..E     |
|      =. +.      |
|       o+        |
+----[SHA256]-----+
Connection check failed.
The primary GitHub Enterprise Server appliance must be configured to allow replica access.
Visit http://172.3?.???.???/setup/settings and authorize the following SSH key:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJev???????????????????????YEf1Kvx7AyBAduoMe admin-ssh-key

Run `ghe-repl-setup 172.3?.???.???' once the key has been added to continue replica setup

Use command result key

• ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJev?????????????????wYEf1Kvx7AyBAduoMe admin-ssh-key

Visit Primary & add ssh key

• Visit http://172.3?.???.???/setup/settings and authorize the following SSH key

Run command (Replica VM)

• ghe-repl-setup $PRIMARY_VM_IP_ADDRESS
• ghe-repl-start
• ghe-repl-status

admin@ip-172-3?-???-???:$ ghe-repl-setup 172.3?.???.???
Verifying ssh connectivity with 172.3?.???.??? ...
Connection check succeeded.
Updating Elasticsearch configuration ...
Elasticsearch isn't listening on tcp/9200.
Copying license and settings from primary appliance ...
 --> Importing SSH host keys...
 --> The SSH host keys on this appliance have been replaced to match the primary.
 --> Please run 'ssh-keygen -R 172.3X.XX.XXX; ssh-keygen -R "[172.3X.XX.XXX]:122"' on your client to prevent future ssh warnings.
Copying custom CA certificates from primary appliance ...
Success: Replica mode is configured against 172.3?.???.???.
To disable replica mode and undo these changes, run 'ghe-repl-teardown'.
Run 'ghe-repl-start' to start replicating from the newly configured primary.
 
 
 
admin@ip-172-3?-???-???:$ ghe-repl-start
Verifying ssh connectivity with 172.3?.???.??? ...
Updating configuration...
Validating configuration
Updating configuration for githubtest-testserver-io-primary (172.3?.???.???)
Configuration Updated
Configuration Phase 1
githubtest-testserver-io-primary: Dec 04 05:46:52 Preparing storage device...
githubtest-testserver-io-primary: Dec 04 05:46:54 Updating configuration...
githubtest-testserver-io-primary: Dec 04 05:46:55 Reloading system services...
githubtest-testserver-io-primary: Dec 04 05:47:19 Done!
githubtest-testserver-io-replica: Dec 04 05:46:52 Preparing storage device...
githubtest-testserver-io-replica: Dec 04 05:46:54 Updating configuration...
githubtest-testserver-io-replica: Dec 04 05:46:55 Reloading system services...
githubtest-testserver-io-replica: Dec 04 05:48:11 Done!
Configuration Phase 2
githubtest-testserver-io-replica: Dec 04 05:48:12 Running migrations...
githubtest-testserver-io-replica: Dec 04 05:48:12 Done!
githubtest-testserver-io-primary: Dec 04 05:48:12 Running migrations...
githubtest-testserver-io-primary: Dec 04 05:48:28 Done!
Configuration Phase 3
githubtest-testserver-io-primary: Waiting for services to be active...
githubtest-testserver-io-primary: Dec 04 05:48:47 Reloading application services...
githubtest-testserver-io-primary: Dec 04 05:49:10 Done!
githubtest-testserver-io-replica: Dec 04 05:48:30 Reloading application services...
githubtest-testserver-io-replica: Dec 04 05:50:06 Done!
Finished cluster configuration
Success: replication is running for all services.
Run `ghe-repl-status' to monitor replication health and progress.
 
 
 
admin@ip-172-3?-???-???:$ ghe-repl-status
OK: mysql replication is in sync
OK: redis replication is in sync
OK: elasticsearch cluster is in sync
OK: git replication is in sync
OK: pages replication is in sync
OK: alambic replication is in sync
OK: git-hooks replication is in sync
OK: consul replication is in sync

GHE server version: 2.19.2

License

https://enterprise.github.com/dashboard (Free Trial or Purchase license)

After Sign in, click 'Download' on top of the menu.

Step 1. Download license

Appliance

Step 2. Appliance

Select Cloud vendor & region

I selected AWS & Mumbai region.

AWS 'EC2' Dashboard & Click Launch Instance

Search AMI with AMI ID: ami-06e551001e2d40c1a

Set EC2 and launch 

• Storage add EBS at least 10GB

Install

Read manual 1st.

https://help.github.com/en/enterprise/2.19/admin/installation/installing-github-enterprise-server-on-aws

Need to open 122(ssh), 8443(https) TCP port before start setup

Upload your license file & set password.

Select installation type

• New Install

Route53 add dns

set hostname & Test domain settings

SSL setup & Click Save settings

Finish setup & Click 'Save settings' 

After finish setup and hostname not working. Then use this link

• https://$SERVER_IP_ADDRESS/setup/settings
• Sample:  https://172.3?.???.???:8443/setup/settings