Dump and restore a Neo4j database on kubernetes

Dump and Restore a Neo4j Database on Kubernetes:
A Comprehensive Guide

Introduction:

Neo4j, a popular graph database, plays a pivotal role in managing interconnected data, making it essential to safeguard your data through regular backups and seamless migration between Kubernetes clusters. This blog post presents a comprehensive guide on how to dump and restore a Neo4j database on Kubernetes. We will leverage a bash script to automate the process and ensure data consistency.

Prerequisite: Installing APOC Plugin for Neo4j:

Before proceeding, ensure that the APOC (Awesome Procedures on Cypher) plugin is installed in both the source and destination Neo4j instances. The APOC plugin offers an array of powerful procedures and functions that extend the capabilities of Cypher queries. For a detailed step-by-step guide on installing the APOC plugin, you can refer to our blog post: Link to blog on installing APOC plugin for Neo4j.

Script

#!/bin/bash

required_vars=(
  SOURCE_PROJECT_ID
  SOURCE_CLUSTER_NAME
  SOURCE_NEO4j_PASSWORD
  DESTINATION_PROJECT_ID
  DESTINATION_CLUSTER_NAME
  DESTINATION_NEO4j_PASSWORD
  NEO4J_USER
  REGION
  NEO4J_POD
  NEO4J_NAMESPACE
)

for var in "${required_vars[@]}"; do
  if [ -z "${!var}" ]; then
    echo "Error: $var is not set or is empty."
    exit 1
  fi
done
echo "All Variables found "

# Authenticate with GCP 
gcloud auth login

# Set the source GCP project
gcloud config set project $SOURCE_PROJECT_ID
 

# Authenticate to the GKE cluster
if gcloud container clusters get-credentials $SOURCE_CLUSTER_NAME \
    --region $REGION \
    --project $SOURCE_PROJECT_ID; then
  echo "Connected to $SOURCE_CLUSTER_NAME"
else
  echo "Failed to connect to $SOURCE_CLUSTER_NAME"
  exit 1
fi

filename="backup_$(date +'%d-%m-%Y')"
# connect to cypher shell and create back up file
if kubectl exec $NEO4J_POD -n $NEO4J_NAMESPACE -- cypher-shell -u $NEO4J_USER -p \
  $SOURCE_NEO4j_PASSWORD -a localhost \
  "CALL apoc.export.json.all('${filename}.json', {useTypes:true})";then
  echo "Backup file created"
else
  echo "Backup file creation fail"
  exit 1
fi

# download file to local
kubectl cp neo4j-db/neo4j-db-0:/import/${filename}.json ${filename}.json
echo "Backup file copied to local"

#setting destination project id
gcloud config set project $DESTINATION_PROJECT_ID

#connect destination cluster
if gcloud container clusters get-credentials $DESTINATION_CLUSTER_NAME --region \
	$REGION --project $DESTINATION_PROJECT_ID; then
  echo "Connected to $DESTINATION_CLUSTER_NAME" 
else
  echo "Failed to connect to $DESTINATION_CLUSTER_NAME"
  exit 1
fi
#copy backfile to neo4j
kubectl cp ${filename}.json  neo4j-db/neo4j-db-0:/import/${filename}.json
echo "Copied file to destination neo4j"
#Add contraint

# add the name of labels of node in bellow list
label_list=()

 for label in "${label_list[@]}"; do
 cypher_query="CREATE CONSTRAINT FOR (n:$label) REQUIRE n.neo4jImportId IS UNIQUE;"
 echo "$cypher_query"
 kubectl exec neo4j-db-0 -n neo4j-db -- cypher-shell -u $NEO4J_USER -p \
 $DESTINATION_NEO4j_PASSWORD -a localhost "$cypher_query"
 done

#Upload data
if kubectl exec $NEO4J_POD -n $NEO4J_NAMESPACE -- cypher-shell -u $NEO4J_USER -p \
  $DESTINATION_NEO4j_PASSWORD -a localhost \
  "CALL apoc.import.json('${filename}.json')"; then
  echo "Data Uploaded to neo4j"
else
  echo "Data Upload to neo4j failed"
  exit 1
fi

Step 1: Understanding the Script:

Let's delve into the bash script that facilitates the data transfer:

Variable Declarations:

The script begins by declaring variables representing source and destination configurations, such as project IDs, cluster names, passwords, and regions/zones. It includes a check for required variables to avoid any missing parameters during execution.

Authentication and Connection:

The script authenticates with Google Cloud Platform (GCP) to gain access to the necessary clusters. It connects to both the source and destination Neo4j clusters using the provided configurations.

Creating the Backup:

The script generates a filename based on the current date to store the JSON backup file. It connects to the source Neo4j pod and employs cypher-shell to create a JSON backup file (${filename}.json) of the Neo4j data.

Copying Backup Locally:

The script copies the backup file from the source cluster to the local machine.

Connecting to Destination Cluster:

The script sets the destination project ID. It then connects to the destination Neo4j cluster.

Copying Backup to Destination:

The script transfers the backup file from the local machine to the destination Neo4j pod.

Adding Constraints (Optional):

The script includes commented code to create constraints for specific labels in the Neo4j database. Constraints help ensure data integrity.

Uploading Data to Destination:

Finally, the script uses cypher-shell to import the JSON data from the backup file into the destination Neo4j database.

Step 2: Preparing for the Dump and Restore Process:

Before executing the script, ensure the following:

• Add the required environment variable mentioned in script or set the value of variables in the script.
• Uncomment and modify the constraints creation section if needed.
• Convert multiline commads to single line

Step 3: Executing the Script:

Once the script is ready, execute it on your local machine. The script will automate the following actions:

1. Authenticate and connect to the source Neo4j cluster.
2. Create a JSON backup of the Neo4j data in the source cluster.
3. Copy the backup file to your local machine.
4. Authenticate and connect to the destination Neo4j cluster.
5. Copy the backup file to the destination Neo4j pod.
6. Optionally, create constraints on the destination Neo4j database.
7. Import the data from the backup file into the destination Neo4j database.

Conclusion:

This comprehensive guide provides you with a step-by-step process to dump and restore a Neo4j database on Kubernetes using a bash script. By ensuring that the APOC plugin is installed and leveraging the script's automation, you can effortlessly transfer and backup your Neo4j data while maintaining data integrity. Remember to handle sensitive data securely and test the process in a non-production environment before performing any actions on a production setup.