Setting up a development environment can be a tedious and time consuming chore. Setup documents can become stale and outdated as versions and dependencies change. In this article, we will walk through the steps for automating the creation of a virtual development environment.
If you have ever written software (or worked with a team that does) on a desktop computer, you have probably encountered the following meme:
Attempting to deploy software in different environments can present a myriad of problems. We frequently struggle when development environments are different than staging, test and production machines. Setup, deployment scripts and techniques used to work on a desktop OS (such as windows) are usually quite different than those used when deploying to a production environment running some form of linux.
Another issue frequently faced is the need for different tools and setups for different projects (quite often at the same time). In the past, we would setup our development stack on our host machine. While this approach may work for a single project, differing requirements and versions make it difficult to maintain for multiple projects.
This is where Vagrant comes in. Vagrant configurations are portable. All members of our team can create their development environments from the same configuration ensuring that everyone is testing in the same environment regardless of the type of workstation they are using.
In the first post of this multi-part series, we will cover the steps to creating an automated development environment for use with Wildfly and Keycloak.
Goals
- Creation of the sandbox development environment should be automated
- Developers should be able to attach a debugger to deployed code
- Scripts used to provision the development environment should be usable for provisioning other life cycle environments
Technology Stack
- Vagrant is a tool to create and configure lightweight, reproducible, and portable development environments
- Virtualbox is a free general purpose virtualizer. Vagrant works with many providers but virtualbox is free and available on many platforms
- Ansible is a provisioning tool used to deploy software and configure systems. Ansible is simple, straightforward and easy to use
- Ansible Galaxy is a community hub for sharing roles. We will be using roles for configuring openldap, nginx, ssl and mysql
Development Environment
The development environment we will be creating will consist of
- Ubuntu Trusty is a Debian based linux operating system.
- Keycloak is an Integrated SSO and IDM for browser apps and RESTful web services
- Wildfly is an open source JEE7 application server
- OpenJDK 8 is an open source implementation of the Java SE 8 platform specification
- Openldap is an open source implementation of the Lightweight Directory Access Protocol
- MySql is an open source database server
- Nginx is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP proxy server
Github
Code for this project can be found at https://github.com/SUMGlobal/keycloak_demo/tree/master/vm-keycloak
Installation
Installing Vagrant, Ansible and Virtualbox is straightforward. You can follow the instructions from the README.md
Details
Vagrant
The Vagrantfile is simple
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# -*- mode: ruby -*- # vi: set ft=ruby : # Vagrantfile API/syntax version. Don't touch unless you know what you're doing! VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "ubuntu/trusty64" config.vm.network :private_network, ip: "172.16.0.100" config.vm.synced_folder "./shared", "/shared", create: true config.ssh.forward_agent = true config.vm.provider :virtualbox do |vb| # Use VBoxManage to customize the VM. For example to change memory: vb.customize ["modifyvm", :id, "--cpus", "2"] vb.customize ["modifyvm", :id, "--memory", "4096"] end config.vm.provision "ansible" do |ansible| ansible.extra_vars = { host_ip_address: `ifconfig en0| grep "inet " | awk '{printf "%s", $2}'` } ansible.playbook = "provision/vagrant.yml" end config.vm.provision :shell, :inline => "rm /etc/localtime && ln -s /usr/share/zoneinfo/America/New_York /etc/localtime && ntpdate pool.ntp.org" end |
- A private network is created using the ip address 172.16.0.100. This address is accessible from our sandbox but not outside it
- A shared folder shared is created for ease of use in copying files back and forth. This is not strictly necessary as the project folder is already mounted as the vagrant folder in the vm but changes to the shared folder are set to be ignored by git (see the .gitignore file)
- SSH agent forwarding is set to true. This will allow us to use our existing ssh keys to pull from our github repository
- Since we are installing all of our application software here we beef up the box by adding 2 cpu’s and using 4096 bytes of memory
Ansible Provisioning
[notification type=”alert-info” close=”false” ][icon type=”fa fa-info-circle fa-2x”] Note: In our example, we are using Ansible to provision the linux virtual machine. I like Ansible because of it’s simplicity of use but it does require a linux or mac control machine. If your development staff is using windows, you will need to use a different provisioner such as Puppet or Chef[/notification]
We break up provisioning by creating roles. The roles we use are
|
1 2 3 4 5 6 7 8 9 10 11 12 |
roles: - common-setup - geerlingguy.mysql - openldap_server - wildfly - keycloak - keycloak_wf9_adapter - serenity-db - ldap-data - keycloak-realm - jdauphant.ssl-certs - jdauphant.nginx |
- common-setup: Used to install common utilities like wget, git and the openjdk 8
- geerlingguy.mysql: Ansible galaxy role used to install and configure Mysql. Check the role vars for configuration
- openldap_server: Ansible galaxy role used to install and configure Openldap. Check the role vars for configuration
- wildfly: Downloads and configures the Wildfly server. After downloading and upacking the server this role also makes configuration changes and installs a Mysql datasource
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
- name: update JBOSS_HOME file lineinfile: dest="{{ wildfly_install_dir }}/docs/contrib/scripts/init.d/wildfly.conf" regexp=^JBOSS_HOME line="JBOSS_HOME=\"{{ wildfly_base_dir }}/{{ wildfly_name }}\"" - name: update JBOSS_USER file lineinfile: dest="{{ wildfly_install_dir }}/docs/contrib/scripts/init.d/wildfly.conf" regexp=^JBOSS_USER line="JBOSS_USER={{ wildfly_owner }}" - name: update JBOSS_OPTS file lineinfile: dest="{{ wildfly_install_dir }}/docs/contrib/scripts/init.d/wildfly.conf" regexp=^JBOSS_OPTS line="JBOSS_OPTS=\"-Djboss.bind.address=0.0.0.0 -Djboss.bind.address.management=0.0.0.0 -Djboss.socket.binding.port-offset={{ wildfly_port_offset }}\"" - name: create link for wildfly.conf to /etc/default/wildfly file: src="{{ wildfly_install_dir }}/docs/contrib/scripts/init.d/wildfly.conf" dest=/etc/default/wildfly state=link - name: create a link for wildfly startup file: src="{{ wildfly_install_dir }}/docs/contrib/scripts/init.d/wildfly-init-debian.sh" dest=/etc/init.d/wildfly state=link - name: create a link to wildfly file: src="{{ wildfly_install_dir }}" dest="{{ wildfly_base_dir }}/wildfly" state=link - name: update wildfly modules copy: src="modules/" dest="{{ wildfly_install_dir }}/modules/system/layers/base" |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# until ansible 2.0 #- name: get mysql jdbc driver # maven_artifact: group_id=mysql artifact_id=mysql-connector-java version=5.1.36 dest="{{ wildfly_install_dir }}/standalone/deployments/" # user: "{{ wildfly_owner }}" - name: download mysql driver get_url: url="http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.36/mysql-connector-java-5.1.36.jar" dest="{{ wildfly_install_dir }}/standalone/deployments/" become: yes become_user: "{{ wildfly_owner }}" - name: copy mysql cli to tmp copy: src=mysql.cli dest=/tmp owner="{{ wildfly_owner }}" group="{{ wildfly_group }}" - name: install mysql driver into standalone.xml through cli command: "{{ wildfly_install_dir }}/bin/jboss-cli.sh --file=/tmp/mysql.cli" become: yes become_user: "{{ wildfly_owner }}" |
|
1 2 3 4 5 6 7 8 9 10 11 12 |
embed-server if (outcome == success) of /subsystem=datasources/data-source=serenityDS:read-resource /subsystem=datasources/data-source=serenityDS:remove end-if /subsystem=datasources/data-source=serenityDS:add( \ jndi-name=java:jboss/datasources/serenityDS, \ driver-class=com.mysql.jdbc.Driver, \ driver-name=mysql-connector-java-5.1.36.jar_com.mysql.jdbc.Driver_5_1, \ connection-url="jdbc:mysql://localhost:3306/serenity", \ user-name="firefly", password="bigdamnheroes", \ use-java-context=true) stop-embedded-server |
- keycloak: Downloads and configures the Keycloak server in the same manner as the Wildfly role. This role also makes the necessary changes to use SSL
|
1 2 3 4 5 |
- name: copy install keycloak ssl cli to tmp copy: src=installKeycloakSSL.cli dest=/tmp owner="{{ keycloak_owner }}" group="{{ keycloak_group }}" - name: install keycloak ssl through admin server cli command: "sudo -u {{ keycloak_owner }} {{ keycloak_install_dir }}/bin/jboss-cli.sh --file=/tmp/installKeycloakSSL.cli" |
|
1 2 3 4 5 6 7 8 9 10 11 12 |
embed-server /subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=proxy-address-forwarding,value=true) /subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=redirect-socket,value=proxy-https) if (outcome == success) of /socket-binding-group=standard-sockets/socket-binding=proxy-https:read-resource /socket-binding-group=standard-sockets/socket-binding=proxy-https:remove() end-if stop-embedded-server embed-server /socket-binding-group=standard-sockets/socket-binding=proxy-https:add(port=443) stop-embedded-server |
- keycloak_wf9_adapter: Downloads and copies the additional configuration and modules needed for Wildfly to use Keycloak for authentication
- serenity-db: Configures Mysql db for our sample application. It creates users and imports tables
- ldap-data: Imports a list of data to populate our Openldap server
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
- name: check if serenity.ldif has been imported stat: path="{{ check_dir }}/serenity_ldap.done" register: ldap_check - name: copy ldif file to tmp copy: src=serenity.ldif dest=/tmp - name: import users and groups command: /usr/bin/ldapadd -c -w {{ openldap_server_rootpw }} -x -D {{ openldap_bind_dn }} -f /tmp/serenity.ldif when: ldap_check.stat.exists == False - name: ldap data imported file: path={{ check_dir }}/serenity_ldap.done state=touch |
- keycloak-realm: Imports a sample realm used by our demo application
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
- name: check if realm has been installed stat: path="{{ check_dir }}/keycloak_realm.done" register: keycloak_realm_check - name: copy realm file to tmp template: src=serenity.j2 dest=/tmp/serenity.realm when: keycloak_realm_check.stat.exists == False - name: make sure keycloak is shutdown service: name=keycloak state=stopped when: keycloak_realm_check.stat.exists == False - name: import realm shell: "{{ app_base_dir }}/keycloak/bin/standalone.sh -Dkeycloak.import=/tmp/serenity.realm > /tmp/keycloak_realm_input.log" become: yes become_user: keycloak async: 60 poll: 0 when: keycloak_realm_check.stat.exists == False - name: wait for import to finish wait_for: path="/tmp/keycloak_realm_input.log" search_regex="WFLYSRV0025:" when: keycloak_realm_check.stat.exists == False - name: create admin user shell: "{{ app_base_dir }}/keycloak/bin/add-user-keycloak.sh -r master -u admin -p admin > /tmp/keycloak_user_creation.log" become: yes become_user: keycloak when: keycloak_realm_check.stat.exists == False - name: shutdown keycloak realm import command: "{{ app_base_dir }}/keycloak/bin/jboss-cli.sh --connect -c shutdown" become: yes become_user: keycloak when: keycloak_realm_check.stat.exists == False - name: keycloak realm installed file: path={{ check_dir }}/keycloak_realm.done state=touch |
- jdauphant.ssl-certs: Ansible galaxy role that generates self-signed SSL certificates. This is used by Nginx. Check the role vars for configuration
- jdauphant.nginx: Ansible galaxy role to configure Nginx for our Keycloak and Wildfly servers. Check the role vars for configuration
Running
At this point, you’re ready to bring up the environment and get down to business
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
$ mkdir keycloak-workspace $ cd keycloak-workspace $ git clone git@github.com:SUMGlobal/keycloak_demo.git $ cd keycloak_demo/vm_keycloak $ vagrant up $ vagrant ssh Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-74-generic x86_64) * Documentation: https://help.ubuntu.com/ System information as of Mon Dec 28 16:05:10 UTC 2015 System load: 0.83 Processes: 97 Usage of /: 3.4% of 39.34GB Users logged in: 0 Memory usage: 3% IP address for eth0: 10.0.2.15 Swap usage: 0% IP address for eth1: 172.16.0.100 Graph this data and manage this system at: https://landscape.canonical.com/ Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud 0 packages can be updated. 0 updates are security updates. Last login: Mon Dec 28 11:15:28 2015 from 10.0.2.2 vagrant@vagrant-ubuntu-trusty-64:~$ vagrant@vagrant-ubuntu-trusty-64:~$ ls -l /opt/jboss/ total 8 lrwxrwxrwx 1 root root 31 Dec 28 11:14 keycloak -> /opt/jboss/keycloak-1.9.4.Final drwxr-xr-x 10 keycloak jboss 4096 Dec 8 05:47 keycloak-1.9.4.Final lrwxrwxrwx 1 root root 30 Dec 28 11:13 wildfly -> /opt/jboss/wildfly-10.0.0.Final drwxr-xr-x 10 wildfly jboss 4096 Oct 26 13:15 wildfly-10.0.0.Final vagrant@vagrant-ubuntu-trusty-64:~$ vagrant@vagrant-ubuntu-trusty-64:~$ sudo service keycloak start * Starting WildFly Application Server keycloak [ OK ] vagrant@vagrant-ubuntu-trusty-64:~$ sudo service wildfly start * Starting WildFly Application Server wildfly [ OK ] vagrant@vagrant-ubuntu-trusty-64:~$ |
In my next post, we will use this container to deploy and secure a set of web services.
Social Media