From patchwork Tue Dec 7 08:49:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: You-Sheng Yang X-Patchwork-Id: 1564549 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=Zpxz401d; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4J7Ys95q61z9t1r for ; Tue, 7 Dec 2021 19:50:05 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4J7Ys94F7bz2yPN for ; Tue, 7 Dec 2021 19:50:05 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=Zpxz401d; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42b; helo=mail-pf1-x42b.google.com; envelope-from=vicamo@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=Zpxz401d; dkim-atps=neutral Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4J7YrM40jZz2ypZ for ; Tue, 7 Dec 2021 19:49:23 +1100 (AEDT) Received: by mail-pf1-x42b.google.com with SMTP id z6so12788686pfe.7 for ; Tue, 07 Dec 2021 00:49:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=4QiXl+87OJUJvYae1fgr+aAa8eX5y5CW2AIUysi/64g=; b=Zpxz401d5dyVWfQzujs1JIeKZYKRH2Uxw7dWDh3w9wErb0troqTk8qtVuqiq9xyrZ2 orz0zCjtmqUIUsohju+tWCjExv9XA0zUkpfXPV9XLSQKovZf3UwsNbl9yvH7OeW4HAcc sszlEodo3EN+kJ1GL9vMwyzO158cTaGPBGtMHTctjMue7TH7PV6HSzx3Stlqh3CCuWl7 4WcijhQnr+liAcrHEgNVyTFGd6ODVsxlqQldkjQZKCHHznEPOq4y6Ffx35WiKterJoIG jWkGwMTgev7GH3Jxzq1bbhJJpiAIUZrUDob7geUS+d+K84Kasuk96Fa9xP4LjdbKo0Js vJVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4QiXl+87OJUJvYae1fgr+aAa8eX5y5CW2AIUysi/64g=; b=fBdWdK/fDzxAgYoxqV9+7FaZ7bFonwYrSL0hkGuDzZb4opLAf1kSbDZ/kLEyqlyi2U KihucJgaj0Gq8GnUWwR9QxMb02EGIW4vcMd/kx2mGNESHwi7Gf4JPiiHjif4EOdR2ir2 7xI/+tPL8wujURsEQ3qJz3SWiyxBJSs27MScKaHlqDkxdpEokKfDm7nAtpc7y4frVT25 vFpt7/oM4YsgiDV1ulPHjCJ24GocZBigNp+eNtXZIHHFu6O5wdwyhmJwhn7HNMDZTVUf u8U4y80zLQW3iyU4nlZ3AVPx52qalnk5EA39jBuxKlyIiI8P2b/x6Je+bZe/ZC039CZv p2HA== X-Gm-Message-State: AOAM532CFgf36M/6wlAB9LrSzKMU9IruHRGeMlT1dYn721/01HrUK2bg 8IvRong9L+xDSIYp5Da/EJyHfugZVPg= X-Google-Smtp-Source: ABdhPJz/OEAsS+DM4q4SEgf41JuOhko9DKZBZ1yPT57NAOBWkq+GI9SyzAoyn2ivB14K54u0DfUIVA== X-Received: by 2002:a63:3605:: with SMTP id d5mr22579060pga.185.1638866960759; Tue, 07 Dec 2021 00:49:20 -0800 (PST) Received: from localhost (218-173-141-247.dynamic-ip.hinet.net. [218.173.141.247]) by smtp.gmail.com with ESMTPSA id u3sm15999812pfk.32.2021.12.07.00.49.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Dec 2021 00:49:20 -0800 (PST) From: You-Sheng Yang To: patchwork@lists.ozlabs.org Subject: [PATCH 1/4] settings: unify database connection variables Date: Tue, 7 Dec 2021 16:49:13 +0800 Message-Id: <20211207084916.240554-2-vicamo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211207084916.240554-1-vicamo@gmail.com> References: <20211207084916.240554-1-vicamo@gmail.com> MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Signed-off-by: You-Sheng Yang --- .github/workflows/ci.yaml | 39 +++++++++++------- docker-compose-pg.yml | 12 +++--- docker-compose.yml | 12 +++--- docs/deployment/installation.rst | 2 +- docs/development/installation.rst | 13 +++--- patchwork/settings/base.py | 39 ++++++++++++++++++ patchwork/settings/dev.py | 28 +------------ patchwork/settings/production.example.py | 17 -------- tools/docker/entrypoint.sh | 51 ++++++++++++++++-------- tox.ini | 4 +- 10 files changed, 121 insertions(+), 96 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index f4a33b3..04c67cb 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -25,13 +25,18 @@ jobs: matrix: python: [3.6, 3.7, 3.8, 3.9] db: [postgres, mysql] + env: + DATABASE_NAME: patchwork + DATABASE_USER: patchwork + DATABASE_PASS: patchwork + MYSQL_ROOT_PASSWORD: root services: postgres: image: postgres:latest env: - POSTGRES_DB: patchwork - POSTGRES_PASSWORD: patchwork - POSTGRES_USER: patchwork + POSTGRES_DB: ${{ env.DATABASE_NAME }} + POSTGRES_PASSWORD: ${{ env.DATABASE_PASS }} + POSTGRES_USER: ${{ env.DATABASE_USER }} ports: - 5432:5432 options: >- @@ -42,10 +47,10 @@ jobs: mysql: image: mysql:latest env: - MYSQL_DATABASE: patchwork - MYSQL_USER: patchwork - MYSQL_PASSWORD: patchwork - MYSQL_ROOT_PASSWORD: root + MYSQL_DATABASE: ${{ env.DATABASE_NAME }} + MYSQL_USER: ${{ env.DATABASE_USER }} + MYSQL_PASSWORD: ${{ env.DATABASE_PASS }} + MYSQL_ROOT_PASSWORD: ${{ env.MYSQL_ROOT_PASSWORD }} ports: - 3306:3306 options: >- @@ -64,22 +69,26 @@ jobs: run: python -m pip install tox tox-gh-actions codecov - name: Log database configuration (mysql) if: ${{ matrix.db == 'mysql' }} - run: mysql -h 127.0.0.1 -e "SELECT VERSION(), CURRENT_USER();" -uroot -proot patchwork + run: | + mysql -h 127.0.0.1 -e "SELECT VERSION(), CURRENT_USER();" \ + -uroot -p${MYSQL_ROOT_PASSWORD} ${DATABASE_NAME} - name: Log database configuration (postgres) if: ${{ matrix.db == 'postgres' }} - run: psql -h 127.0.0.1 -c "SELECT VERSION(), CURRENT_USER, current_database()" -U patchwork -d patchwork + run: | + psql -h 127.0.0.1 -c "SELECT VERSION(), CURRENT_USER, current_database()" \ + -U ${DATABASE_USER} -d ${DATABASE_NAME} env: - PGPASSWORD: patchwork + PGPASSWORD: ${{ env.DATABASE_PASS }} - name: Modify database user permissions (mysql) if: ${{ matrix.db == 'mysql' }} - run: mysql -h 127.0.0.1 -e "GRANT ALL ON \`test\\_patchwork%\`.* to 'patchwork'@'%';" -uroot -proot + run: | + mysql -h 127.0.0.1 -e "GRANT ALL ON \`test\\_${DATABASE_NAME}%\`.* to '${DATABASE_USER}'@'%';" \ + -uroot -p${MYSQL_ROOT_PASSWORD} - name: Run unit tests (via tox) run: tox env: - PW_TEST_DB_TYPE: "${{ matrix.db }}" - PW_TEST_DB_USER: "patchwork" - PW_TEST_DB_PASS: "patchwork" - PW_TEST_DB_HOST: "127.0.0.1" + DATABASE_TYPE: "${{ matrix.db }}" + DATABASE_HOST: "127.0.0.1" docs: name: Build docs runs-on: ubuntu-latest diff --git a/docker-compose-pg.yml b/docker-compose-pg.yml index 44c49fb..a45240e 100644 --- a/docker-compose-pg.yml +++ b/docker-compose-pg.yml @@ -24,9 +24,9 @@ services: environment: - UID - GID - - PW_TEST_DB_HOST=db - - PW_TEST_DB_PORT=5432 - - PW_TEST_DB_TYPE=postgres - - PW_TEST_DB_USER=postgres - - PW_TEST_DB_PASS=password - - PGPASSWORD=password + - DATABASE_TYPE=postgres + - DATABASE_HOST=db + - DATABASE_PORT=5432 + - DATABASE_NAME=patchwork + - DATABASE_USER=postgres + - DATABASE_PASS=password diff --git a/docker-compose.yml b/docker-compose.yml index 1d49c51..1673823 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -5,9 +5,7 @@ services: volumes: - ./tools/docker/db/data:/var/lib/mysql environment: - - MYSQL_ROOT_PASSWORD=password - - MYSQL_USER=patchwork - - MYSQL_PASSWORD=password + - MYSQL_ROOT_PASSWORD=root web: build: @@ -26,5 +24,9 @@ services: environment: - UID - GID - - PW_TEST_DB_HOST=db - - PW_TEST_DB_PORT=3306 + # skip DATABASE_TYPE explicitly as mysql should be the default type. + - DATABASE_HOST=db + - DATABASE_PORT=3306 + - DATABASE_USER=patchwork + - DATABASE_PASS=password + - MYSQL_ROOT_PASSWORD=root diff --git a/docs/deployment/installation.rst b/docs/deployment/installation.rst index 1abf187..9df31d9 100644 --- a/docs/deployment/installation.rst +++ b/docs/deployment/installation.rst @@ -229,7 +229,7 @@ As a reminder, these were: - ``DATABASE_NAME`` - ``DATABASE_USER`` -- ``DATABASE_PASSWORD`` +- ``DATABASE_PASS`` - ``DATABASE_HOST`` - ``DATABASE_PORT`` diff --git a/docs/development/installation.rst b/docs/development/installation.rst index d16177c..59c7847 100644 --- a/docs/development/installation.rst +++ b/docs/development/installation.rst @@ -316,7 +316,7 @@ if necessary: The ``patchwork`` username and ``password`` password are the defaults expected by the provided ``dev`` settings files. If using something - different, export the ``PW_TEST_DB_USER`` and ``PW_TEST_DB_PASS`` variables + different, export the ``DATABASE_USER`` and ``DATABASE_PASS`` variables described in the :ref:`Environment Variables ` section below. Alternatively, you can create your own settings file with these variables hardcoded and change the value of ``DJANGO_SETTINGS_MODULE`` as described @@ -470,17 +470,16 @@ __ https://django-dbbackup.readthedocs.io/en/stable/ Environment Variables --------------------- -The following environment variables are available to configure settings when -using the provided ``dev`` settings file. +The following environment variables are available to configure settings. -``PW_TEST_DB_NAME=patchwork`` +``DATABASE_NAME=patchwork`` Name of the database -``PW_TEST_DB_USER=patchwork`` +``DATABASE_USER=patchwork`` Username to access the database with -``PW_TEST_DB_PASS=password`` +``DATABASE_PASS=password`` Password to access the database with< -``PW_TEST_DB_TYPE=mysql`` +``DATABASE_TYPE=mysql`` Type of database to use. Options: ``mysql``, ``postgres`` diff --git a/patchwork/settings/base.py b/patchwork/settings/base.py index ff14d91..d71e7dd 100644 --- a/patchwork/settings/base.py +++ b/patchwork/settings/base.py @@ -105,6 +105,45 @@ STATICFILES_DIRS = [ os.path.join(ROOT_DIR, 'htdocs'), ] +# Database +# +# If you're using a postgres database, connecting over a local unix-domain +# socket, then the following setting should work for you. Otherwise, +# see https://docs.djangoproject.com/en/2.2/ref/settings/#databases + +if os.getenv('DATABASE_TYPE', None) == 'postgres': + DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.postgresql_psycopg2', + 'HOST': os.environ.get('DATABASE_HOST', 'localhost'), + 'PORT': os.environ.get('DATABASE_PORT', ''), + 'NAME': os.environ.get('DATABASE_NAME', 'patchwork'), + 'USER': os.environ.get('DATABASE_USER', 'patchwork'), + 'PASSWORD': os.environ.get('DATABASE_PASS', 'password'), + }, + } +elif os.getenv('DATABASE_TYPE', None) == 'sqlite3': + DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': os.environ.get('DATABASE_NAME', ''), + }, + } +else: + DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.mysql', + 'HOST': os.getenv('DATABASE_HOST', 'localhost'), + 'PORT': os.getenv('DATABASE_PORT', ''), + 'NAME': os.getenv('DATABASE_NAME', 'patchwork'), + 'USER': os.getenv('DATABASE_USER', 'patchwork'), + 'PASSWORD': os.getenv('DATABASE_PASS', 'password'), + 'TEST': { + 'CHARSET': 'utf8', + }, + }, + } + # # Third-party application settings # diff --git a/patchwork/settings/dev.py b/patchwork/settings/dev.py index 2b7fc95..cb4cb19 100644 --- a/patchwork/settings/dev.py +++ b/patchwork/settings/dev.py @@ -7,8 +7,6 @@ Design based on: http://www.revsys.com/blog/2014/nov/21/recommended-django-project-layout/ """ -import os - from .base import * # noqa try: @@ -36,30 +34,8 @@ SECRET_KEY = '00000000000000000000000000000000000000000000000000' # noqa DEBUG = True -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.mysql', - 'HOST': os.getenv('PW_TEST_DB_HOST', 'localhost'), - 'PORT': os.getenv('PW_TEST_DB_PORT', ''), - 'USER': os.getenv('PW_TEST_DB_USER', 'patchwork'), - 'PASSWORD': os.getenv('PW_TEST_DB_PASS', 'password'), - 'NAME': os.getenv('PW_TEST_DB_NAME', 'patchwork'), - 'TEST': { - 'CHARSET': 'utf8', - }, - }, -} - -if os.getenv('PW_TEST_DB_TYPE', None) == 'postgres': - DATABASES['default']['ENGINE'] = 'django.db.backends.postgresql_psycopg2' - DATABASES['default']['HOST'] = os.getenv('PW_TEST_DB_HOST', '') -elif os.getenv('PW_TEST_DB_TYPE', None) == 'sqlite': - DATABASES['default']['ENGINE'] = 'django.db.backends.sqlite3' - DATABASES['default']['NAME'] = '/dev/shm/patchwork.test.db.sqlite3' - del DATABASES['default']['HOST'] - del DATABASES['default']['PORT'] - del DATABASES['default']['USER'] - del DATABASES['default']['PASSWORD'] +if DATABASES['default']['ENGINE'] == 'mysql': # noqa: F405 + DATABASES['default']['TEST'] = {'CHARSET': 'utf8'} # noqa: F405 EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend' diff --git a/patchwork/settings/production.example.py b/patchwork/settings/production.example.py index caaf429..ff05629 100644 --- a/patchwork/settings/production.example.py +++ b/patchwork/settings/production.example.py @@ -48,23 +48,6 @@ ADMINS = ( # ('Jeremy Kerr', 'jk@ozlabs.org'), ) -# Database -# -# If you're using a postgres database, connecting over a local unix-domain -# socket, then the following setting should work for you. Otherwise, -# see https://docs.djangoproject.com/en/2.2/ref/settings/#databases - -DATABASES = { - 'default': { - 'ENGINE': 'django.db.backends.postgresql_psycopg2', - 'NAME': os.environ.get('DATABASE_NAME', ''), - 'USER': os.environ.get('DATABASE_USER', ''), - 'PASSWORD': os.environ.get('DATABASE_PASSWORD', ''), - 'HOST': os.environ.get('DATABASE_HOST', ''), - 'PORT': os.environ.get('DATABASE_PORT', ''), - }, -} - # # Static files settings # https://docs.djangoproject.com/en/2.2/ref/settings/#static-files diff --git a/tools/docker/entrypoint.sh b/tools/docker/entrypoint.sh index 5450a53..d111944 100755 --- a/tools/docker/entrypoint.sh +++ b/tools/docker/entrypoint.sh @@ -1,45 +1,62 @@ #!/bin/bash set -euo pipefail -PW_TEST_DB_TYPE=${PW_TEST_DB_TYPE:-mysql} +export DATABASE_HOST=${DATABASE_HOST:-} +export DATABASE_PORT=${DATABASE_PORT:-} +export DATABASE_NAME=${DATABASE_NAME:-patchwork} +export DATABASE_USER=${DATABASE_USER:-patchwork} +export DATABASE_PASS=${DATABASE_PASS:-password} + +case "${DATABASE_TYPE:-}" in +postgres) + export PGPORT=${DATABASE_PORT} + export PGPASSWORD=${DATABASE_PASS} + psql_args=( ${DATABASE_HOST:+--host=${DATABASE_HOST}} "--username=${DATABASE_USER}" ) + ;; +*) + export DATABASE_TYPE=mysql + mysql_args=( ${DATABASE_HOST:+--host=${DATABASE_HOST}} ${DATABASE_PORT:+--port=${DATABASE_PORT}} "--user=${DATABASE_USER}" "--password=${DATABASE_PASS}" ) + mysql_root_args=( ${DATABASE_HOST:+--host=${DATABASE_HOST}} ${DATABASE_PORT:+--port=${DATABASE_PORT}} "--user=root" "--password=${MYSQL_ROOT_PASSWORD:-}" ) + ;; +esac # functions test_db_connection() { - if [ ${PW_TEST_DB_TYPE} = "postgres" ]; then - echo ';' | psql -h $PW_TEST_DB_HOST -U postgres 2> /dev/null > /dev/null + if [ ${DATABASE_TYPE} = "postgres" ]; then + echo ';' | psql "${psql_args[@]}" 2> /dev/null > /dev/null else - mysqladmin -h $PW_TEST_DB_HOST -u patchwork --password=password ping > /dev/null 2> /dev/null + mysqladmin "${mysql_root_args[@]}" ping > /dev/null 2> /dev/null fi } test_database() { - if [ ${PW_TEST_DB_TYPE} = "postgres" ]; then - echo ';' | psql -h $PW_TEST_DB_HOST -U postgres patchwork 2> /dev/null + if [ ${DATABASE_TYPE} = "postgres" ]; then + echo ';' | psql "${psql_args[@]}" "${DATABASE_NAME}" 2> /dev/null else - echo ';' | mysql -h $PW_TEST_DB_HOST -u patchwork -ppassword patchwork 2> /dev/null + echo ';' | mysql "${mysql_args[@]}" "${DATABASE_NAME}" 2> /dev/null fi } reset_data_mysql() { - mysql -uroot -ppassword -h $PW_TEST_DB_HOST << EOF -DROP DATABASE IF EXISTS patchwork; -CREATE DATABASE patchwork CHARACTER SET utf8; -GRANT ALL ON patchwork.* TO 'patchwork' IDENTIFIED BY 'password'; -GRANT ALL ON \`test\\_patchwork%\`.* to 'patchwork'@'%'; + mysql "${mysql_root_args[@]}" << EOF +DROP DATABASE IF EXISTS ${DATABASE_NAME}; +CREATE DATABASE ${DATABASE_NAME} CHARACTER SET utf8; +GRANT ALL ON ${DATABASE_NAME}.* TO '${DATABASE_USER}' IDENTIFIED BY '${DATABASE_PASS}'; +GRANT ALL ON \`test\\_${DATABASE_NAME}%\`.* to '${DATABASE_USER}'@'%'; FLUSH PRIVILEGES; EOF } reset_data_postgres() { - psql -h $PW_TEST_DB_HOST -U postgres <