Automate DBA Tasks With Ansible

Transcription

1 Automate DBA Tasks With Ansible Automation Ivica Arsov October 19, 2017

2 Ivica Arsov Database Consultant Oracle Certified Master 12c & 11g Oracle ACE Associate Blogger Twitter: IvicaArsov Blog: (c) The Pythian Group Inc.,

3 ABOUT PYTHIAN Pythian s 400+ IT professionals help companies adopt and manage disruptive technologies to better compete (c) The Pythian Group Inc.,

4 PYTHIAN MAKES DIGITAL TRANSFORMATION EASIER AND IT OPERATIONS MORE RELIABLE Expert disruptors for innovation and transformation Trusted protectors for ongoing delivery

5 TECHNICAL EXPERTISE Big Data: Harnessing the transformative power of data on a massive scale Advanced Analytics: Mining data for insights & business transformation using data science Databases: Ensuring databases are reliable, secure, available and continuously optimized Cloud: Using the disruptive nature of cloud for accelerated, cost-effective growth Infrastructure: Transforming and managing the IT infrastructure that supports the business DevOps: Providing critical velocity in software deployment by adopting DevOps practices (c) The Pythian Group Inc.,

6 AGENDA Introduction to Ansible Installation Playbooks Roles Templates Modules Custom modules (c) The Pythian Group Inc.,

7 first things first why automation? some considerations Manage multiple servers (> 20-30) Multiple people are doing same thing differently Different configurations Forgotten steps/checks for long action plans Automation reduces human errors etc (c) The Pythian Group Inc.,

8 Ansible introduction and installation

9 ansible engine Open - source automation engine Owned by RedHat Available for Red Hat Enterprise Linux, CentOS, Debian, Ubuntu, OEL Windows support - only as a target machine Written in Python Agentless architecture Git repository: (c) The Pythian Group Inc.,

10 ansible consist of architecture Inventory configuration Modules Playbooks Roles (c) The Pythian Group Inc.,

11 ansible architecture architecture Two types of servers: controlling machines and nodes (targets) Targets are managed by a controlling machine over SSH, WINRM database servers control machine orchestrates (SSH, WINRM) web servers no agents (c) The Pythian Group Inc.,

12 ansible execution The modules are copied and executed on target machines Destination can be controlled with local_tmp in ansible.cfg Execution steps: 1. Create local temporary directory: $HOME/.ansible/tmp/ansible-tmp-xxx/ansiballz_cache 2. Copy module 3. Execute module 4. Return result in JSON format 5. Clear (remove) ansible-tmp-xxx/ansiballz_cache (c) The Pythian Group Inc.,

13 ansible installation RPMs for Enterprise Linux 6, 7 are available from yum via EPEL Add EPEL on OEL7, RHEL7 or CentOS wget yum install epel-release-latest-7.noarch.rpm Install Ansible yum install ansible (c) The Pythian Group Inc.,

14 ansible installation OSX Installation is done via pip Install Xcode Install pip and ansible sudo easy_install pip pip install ansible For more information see: (c) The Pythian Group Inc.,

15 ansible configuration files Ansible configuration file: /etc/ansible/ansible.cfg Define the hosts in a hosts file, by default in /etc/ansible/hosts List all defined hosts: ansible all --list-hosts... db8.example.org [webservers] app1.example.org app2.example.org [dbservers] db1.example.org ansible_host=... ansible_port=... ansible_ssh_private_key_file=... ansible_user=... db2.example.org (c) The Pythian Group Inc.,

16 ansible windows support Control machine requires pywinrm, a Python module for the Windows Remote Management (WinRM) Option Local Accounts Active Directory Accounts Credential Delegation Basic Yes No No Certificate Yes No No Kerberos No Yes Yes NTLM Yes Yes No CredSSP Yes Yes Yes (c) The Pythian Group Inc.,

17 ansible windows support example configuration with kerberos Kerberos configuration file /etc/krb5.conf... [realms] WINGROUP.AD = { kdc = win-hrms-srv.wingroup.ad admin_server = win-hrms-srv.wingroup.ad default_domain = WINGROUP.AD } [domain_realm].wingroup.ad = WINGROUP.AD wingroup.ad = WINGROUP.AD (c) The Pythian Group Inc.,

18 ansible windows support hosts file Inventory/hosts... [windows] win-hrms-srv.wingroup.ad [windows:vars] ansible_user = iarsov@wingroup.ad ansible_password = ******** ansible_connection = winrm ansible_port = 5986 ansible_winrm_server_cert_validation =ignore (c) The Pythian Group Inc.,

19 Demo Ansible installation

20 ad-hoc task execution The ping example Ad-hoc tasks are run with ansible command Ansible uses SSH authentication, you need to specify an user target module user ansible vm12r1 -m ping -u root -k SSH password: vm12r1 SUCCESS => { "changed": false, "ping": "pong" } (c) The Pythian Group Inc.,

21 Playbooks

22 YAML basic rules YAML Ain't Markup Language Human readable data structure Less complex than JSON or XML YAML is case sensitive Does not allow tabs. You must use spaces. YAML db_list: - oracle_owner: oracle oracle_home: /u01/app... sid: orcl... XML <db_list> <oracle_owner>oracle</oracle_owner> <oracle_home>/u01/app...</oracle_home> <sid>orcl</sid>... </db_list> (c) The Pythian Group Inc.,

23 playbooks Organize the play tasks Play example: Playbook s language is YAML playbook.yml - hosts: dbservers Each play contains set of tasks Tasks are executed one at a time in order against all matched targets user: root tasks: - name: ping hostname ping: - name: create directory file: path=/home/ansible/poug2017 state=directory Command ansible-playbook (c) The Pythian Group Inc.,

24 playbooks - hosts: dbservers user: root tasks: - name: ping hostname Playbooks can contain multiple plays Split tasks performed per different host groups ping: - hosts: webservers user: root tasks: - name: create directory file: path=/home/ansible/poug2017 state=directory (c) The Pythian Group Inc.,

25 variables using variables Variables are used using the Jinja2 templating system Variables file definition (var_def.yml) hostname: ansible-demo dir_path: /home/ansible/poug2017 You can also use variables in Templates user: root Variables are referenced with double curly brackets: {{ variable_name }} Playbook file definition - hosts: {{ hostname }} user: {{ user }} vars_files: - var_def.yml tasks: - name: create directory file: path={{ dir_path }} state=directory (c) The Pythian Group Inc.,

26 variables Variables can be defined within the playbook - hosts: {{ hostname }} user: {{ user }} tasks: - name: create directory file: path={{ dir_path }} state=directory vars: - hostname: ansible-demo user: root dir_path: /home/ansible/poug2017 (c) The Pythian Group Inc.,

27 variables Variables can also be defined from command line ansible-playbook demo2.yml -k --extra-vars="dir_path=/home/ansible/poug2017 user=root hostname=ansible-demo SSH password: PLAY [ansible-demo] *************************************************************************************** ok: [ansible-demo] TASK [create directory] *************************************************************************************** ok: [ansible-demo] PLAY RECAP *************************************************************************************** ansible-demo : ok=2 changed=0 unreachable=0 failed=0 (c) The Pythian Group Inc.,

28 ansible windows support Playbook wintest.yml name: Windows demo hosts: windows tasks: - name: Create directory "D:\ansible\poug2017" win_command: Powershell.exe "mkdir D:\ansible\poug2017" (c) The Pythian Group Inc.,

29 ansible windows support playbook execution ansible-playbook wintest.yml PLAY [Windows demo] **************************************************************** TASK [Gathering Facts] ************************************************************* ok: [win-hrms-srv.wingroup.ad] TASK [Create directory D:\ansible\poug2017"] ************************************** changed: [win-hrms-srv.wingroup.ad] PLAY RECAP ************************************************************************* win-hrms-srv.wingroup.ad : ok=2 changed=1 unreachable=0 failed=0 (c) The Pythian Group Inc.,

30 Roles and Templates

31 roles how to organize the tasks? Represent automation within ansible You can define variables for all roles or per role Not complex at all. No hidden magic. Supports automatic load of main.yml file for tasks, handlers, variables, meta definitions and role default variables (c) The Pythian Group Inc.,

32 project structure dbservers role site.yml dbswinstall.yml roles/ dbinstall/ files/ templates/ tasks/ main.yml handlers/ vars/ main.yml playbook role tasks definitions variable definitions dbswinstall.yml snippet: hosts: chicago user: oracle roles: - hostconfig - dbinstall - dbmonitor defaults/... *main.yml files are automatically loaded (c) The Pythian Group Inc.,

33 role dependencies Role dependencies are defined in meta/main.yml file Allows you to reference other roles before running a role... dbswinstall.yml roles/ common/ dbinstall/ files/ meta/ main.yml tasks/ --- dependencies: - { role: common }... (c) The Pythian Group Inc.,

34 Templates Introduction to ansible templating Sometimes pre-defined configuration files are needed for installation An example is the Oracle dbca configuration file Templating allows us to use ansible variables substation within templates (c) The Pythian Group Inc.,

35 Templates... # # Specify the hostname of the system as set during the install. It can be used # to force the installation to use an alternative hostname rather than using the # first hostname found on the system. (e.g., for systems with multiple hostnames # and network interfaces) # ORACLE_HOSTNAME={{ ansible_hostname }} # # Specify the Unix group to be set for the inventory directory. # UNIX_GROUP_NAME={{ oracle_group }}... (c) The Pythian Group Inc.,

36 Templates Source template file is copied to target destination and variables are substituted - name: create 11g installer response file template: src: db_11204.rsp.j2 dest: /tmp/db_11204.rsp owner: "{{ oracle_user }}" group: "{{ oracle_group }}" mode: 0640 (c) The Pythian Group Inc.,

37 Demo Example: Oracle software installation

38 oracle database install installation of oracle oracle_base: "/oracle/app/oracle" oracle_home: "/oracle/app/oracle/product/ /db1" oracle_user: oracle oracle_group: dba swlib_path: "/oracle/install" db_edition: EE DBComponents: "oracle.rdbms.partitioning: " (c) The Pythian Group Inc.,

39 oracle database install installation of oracle name: create oracle base directory file: path: "{{ oracle_base }}" state: directory owner: "{{ oracle_user }}" group: "{{ oracle_group }}" mode: 0775 (c) The Pythian Group Inc.,

40 oracle database install installation of oracle name: create 11g installer response file template: src: db_11204.rsp.j2 dest: /tmp/db_11204.rsp owner: "{{ oracle_user }}" group: "{{ oracle_group }}" mode: 0640 (c) The Pythian Group Inc.,

41 continued installation of oracle name: install base software command: "{{ swlib_path }}/11204/installer/database/runInstaller -silent -ignoreprereq -ignoresysprereqs -waitforcompletion -responsefile /tmp/db_11204.rsp" register: install_db_software args: creates: "{{ oracle_home }}" failed_when: "'skipped' not in install_db_software.stdout and 'Successfully Setup Software.' not in install_db_software.stdout (c) The Pythian Group Inc.,

42 continued installation of oracle name: run root-script orainstroot.sh command: "{{ oracle_base }}/../orainventory/orainstroot.sh" when: "'skipped' not in install_db_software.stdout" become: true become_user: root become_method: sudo (c) The Pythian Group Inc.,

43 continued installation of oracle name: run root-script root.sh from ORACLE_HOME command: "{{ oracle_home }}/root.sh" when: "'skipped' not in install_db_software.stdout" become: true become_user: root become_method: sudo - name: clean up installer response file file: path: /tmp/db_11204.rsp state: absent (c) The Pythian Group Inc.,

44 Modules

45 modules Library plugins, always are executed on target machine Ansible comes with large number of modules Each module supports specific number of arguments target module argument argument ansible vm12r1 -m copy -a "src=/tmp/file.txt dest=/tmp/file.txt" SSH password: vm12r1 SUCCESS => { "changed": true, "checksum": "da39a3ee5e6b4b0d3255bfef afd80709", "dest": "/tmp/file.txt", "gid": 500, "group": "dba",... } (c) The Pythian Group Inc.,

46 custom modules developing modules First check if similar module already exist GitHub (all module updates): Not enough documentation regarding development If you want additional checks on control machine use action_plugins (c) The Pythian Group Inc.,

47 our first custom module developing modules Test module date.py code ansible ansible-demo --module-path=/home/ansible -m date ansible-demo SUCCESS => { "changed": false, "time": " :48: } #!/usr/bin/python import datetime import json date = str(datetime.datetime.now()) print(json.dumps({ "time" : date })) (c) The Pythian Group Inc.,

48 orapatch custom module Can be used to apply PSU on 10g, 11g and 12c In summary: You need to specify: oracle home path, PSU or DBBP patch ID, whether to patch only the oracle home binaries, all databases or specific set of databases It will automatically: stop OH services, apply PSU or DBBP according specified settings, start back the services as before the patching (c) The Pythian Group Inc.,

49 orapatch workflow Check conflicts against OH Patch DB OJVM always executed Push SQL scripts Patch OH End logger session Check OPatch minimum version Patch OH OJVM Start logger session Patch DB Fetch orapatch logfile (c) The Pythian Group Inc.,

50 orapatch vars/main.yml db_list: - oracle_owner: oracle oracle home to patch oracle_home_path: /oracle/app/oracle/product/ /db1 patch_directory: " /psu" directory where patch binaries are located run_only_checks: False run only prerequisite checks patch: True patch_id: patch only oracle specified oracle home patch_only_oh: True apply OJVM PSU patch_ojvm: False patch all databases for specified OH patch_db_all: False patch specific databases patch_db_list: - '{ "dbname": "orcl" }' (c) The Pythian Group Inc.,

51 patch_dict: : patch_proactive_bp_id: patch_gi_id: patch_db_id: patch_ocw_id: patch_ojvm_id: patch_acfs_id: patch_dbwlm_id: patch_dir: file: p _121020_linux-x86-64.zip only_oh: False desc: "DATABASE PROACTIVE BUNDLE PATCH : patch_proactive_bp_id: patch_gi_id: (c) The Pythian Group Inc.,

52 - name: Patch oracle software serial: 1 vars_prompt: - name: "root_password" prompt: "Enter root password (press enter to skip)" private: yes - name: "root_password_confirm" prompt: "Enter root password again (press enter to skip)" private: yes pre_tasks: - assert: that: root_password == root_password_confirm msg: "Root password missmatch." hosts: database user: oracle roles: - orapatch (c) The Pythian Group Inc.,

53 (c) The Pythian Group Inc.,

54 (c) The Pythian Group Inc.,

55 (c) The Pythian Group Inc.,

56 (c) The Pythian Group Inc.,

57 (c) The Pythian Group Inc.,

58 (c) The Pythian Group Inc.,

59 ansible blog posts AUTOMATINC ORACLE RMAN BACKUP CONFIGURATION ON LINUX WITH ANSIBLE CREATING ANSIBLE CUSTOM MODULE FOR AWR REPORT GENERATION SIMPLE STEPS TO PERFORM OPATCH MAINTENANCE WITH ANSIBLE ANSIBLE AND AWS AUTOMATION More at: (c) The Pythian Group Inc.,

60 Thank you. Questions? (c) The Pythian Group Inc.,