Hands-on-4: Name Service and Database Partitioning

All files of this hands-on can be downloaded from THIS LINK.

This hands-on is based on the sock shop we deployed and optimized in hands-on 3. In this hands-on, we are going to further optimize the architecture of the website for easier management and better performance.

1 File Structure

This hands-on contains the following files:

handson4
    |--dockerfiles
        |--XX.dockerfile

• dockerfiles: containing docker files of all docker images you need to deploy in this hands-on. Just for reference

2 Configuration Management

In hands-on 3, we have deployed the website using docker containers. However, as you may have suffered, the docker command-line based naming service using --add-host is very inconvenient and inflexible. For example, to redeploy a component with another IP address, redeployment of all depending components is needed.

Therefore, in this section, we are going to implement another reliable naming service based on Apache Zookeeper cluster.

Question 1: What's relationship between Apache Zookeeper and paxos protocol?

The architecture of the naming service looks like:

• Container: one of the docker containers deployed in Lab3

• • Microservice: the microservice running inside the docker container
  • /etc/hosts: host file of the container. Containing IP to name mappings for other components.
  • Zookeeper Watcher: a small program registering self to the Zookeeper Cluster and monitoring updates in the Zookeeper Cluster. When name mappings in Zookeeper Cluster have changed, update the host file accordingly.

• Zookeeper Cluster: A cluster of Zookeeper instances deployed using docker containers. It is used to store the name to IP address mapping.

All components in the sock shop resolve the IP address of other components according to the container's host file. In previous hands-on, all name mappings in/etc/hosts are added by hard-coded --add-host parameter in docker run command line. Now, we are going to store those mappings in Zookeeper Cluster and write a small program to update the host file according to the cluster automatically.

Please follow those steps to implement the naming service:

Step 1: Deploy the Zookeeper cluster

A Zookeeper cluster is needed to store the configuration data reliably. To store configuration data, one single Zookeeper instance is enough. However, for reliability and availability consideration, multiple replicated Zookeeper instances are needed. Although distributing instances to different physical machines further improves the reliability and availability, we deploy all instances on one single machine in this hands-on due to resource limitation.

Question 2: Many databases or file systems also provide distributed replication support. However, many real-world applications choose Zookeeper for configurations management. Why? What feature of Zookeeper makes it an excellent choice for configuration management?

Step 2: Write the watcher application

The watcher application has three functionalities: registering, initializing, and monitoring.

• Registering: Once the container starts, the watcher should register the container to the Zookeeper cluster. To do so, the watcher should maintain the container's IP address and store the component's domain name to IP mapping into the Zookeeper cluster.
• Initializing: When the container starts, the system's host file in the container is nearly empty. To connect all components, the watcher thread should retrieve all domain-name-to-IP mappings stored in the Zookeeper cluster and update them to the container's host file.
• Monitoring: While running, the watcher should watch all store operations to the Zookeeper cluster and update the container's host file accordingly.

HINT: Zookeeper has provided its client libraries in two languages: Java and C. For more comfortable developing and distributing, we suggest you use the Java version. The programmer's guide for zookeeper's client can be found HERE, and the ZKClient library can make your development much more straightforward. The IP address and port of the Zookeeper cluster can be passed to the watcher program via environment variables or command line arguments.

Step 3: Add watcher into docker images

The watcher application should be deployed into each microservice's container to update components' host files. To do so, we need to rebuild all docker images by rewriting docker files and build them. The new docker files should inherit from previous docker files using FROM command. Your application files can be added to the docker image using ADD or COPY command. The CMD or ENTRYPOINT field of the docker file should also be overridden to start your application at the container's start-up time.

All docker files of microservices are provided in the dockerfiles folder for your reference. For your convenience, we have also updated all docker images with Java support. The new docker images of each component are listed below. You can choose to inherit from either the original images or our new images with Java support.

Step 4: Enjoy redeploying your application

After all previous steps, you can now redeploy your application again using the new naming service. Although components can register themselves automatically, the deployment order of components should still conform to the dependency relationship to avoid initialization errors.

Question 3: Please implement the naming service by yourself. You need to submit your source codes and all extended docker files in the final answer package and briefly describe your implementation in the document.

3 Database Partition

In hands-on 3, we have only deployed one MongoDB instance to serve all no-SQL database requests. However, this makes the database sometimes a performance bottleneck.

Consider a workload where many users are trying to log into the website while other users are trying to purchase socks. Both types of requests are accessing the same database, which makes them interfere with each other. What's more, too much load on a single database instance may further affect the database's performance.

Therefore, it is a better choice to provide each component with a separated database instance. To do so, we need to deploy three MongoDB instances and assign them with domain names carts-db, user-db, and orders-db for component Cart, User, and Orders. After that, those microservices can directly access separated databases using different domain names.

Unfortunately, the poor implementation of backend microservices makes it hard to achieve the database's maximum throughput in our website. Therefore, no test cases are provided for this part, and we only need to change our deployment.

Question 4: Please partition the database according to the previous description and briefly introduct your solution.

4 Hands in

To hand in, please compress the following files into a compressed file with a title like 516037910xxx.zip.

• Program source code for Question 3 without any executable files or libraries.
• Docker files for Questions 3.
• Deployment scripts for Question 4 (named as deploy.sh).
• Answers for Question 1 - Question 4 (named as answers.pdf).

After that, please upload your compressed file to our public FTP (ftp://public.sjtu.edu.cn/upload/handson4, username: followmoon, password: public) before 2019-01-05 23:59.

Note: Please don't add images with too large file size into your document due to the size limit of our public FTP.