[enh] settings.yml: add use_default_settings option

This change is backward compatible with the existing configurations.

If a settings.yml loaded from an user defined location (SEARX_SETTINGS_PATH or /etc/searx/settings.yml),
then this settings can relied on the default settings.yml with this option:
user_default_settings:True
This commit is contained in:
Alexandre Flament 2020-11-03 15:29:59 +01:00
parent 6ada5bac60
commit 1cfe7f2a75
7 changed files with 312 additions and 29 deletions

View file

@ -266,4 +266,19 @@ test.clean:
travis.codecov: travis.codecov:
$(Q)$(PY_ENV_BIN)/python -m pip install codecov $(Q)$(PY_ENV_BIN)/python -m pip install codecov
.PHONY: $(PHONY)
# user-settings
# -------------
PHONY += user-settings.create user-settings.update
user-settings.update: pyenvinstall
$(Q)$(PY_ENV_ACT); pip install ruamel.yaml
$(Q)$(PY_ENV_ACT); python utils/update_user_settings.py ${SEARX_SETTINGS_PATH}
user-settings.update.engines: pyenvinstall
$(Q)$(PY_ENV_ACT); pip install ruamel.yaml
$(Q)$(PY_ENV_ACT); python utils/update_user_settings.py --add-engines ${SEARX_SETTINGS_PATH}
.PHONY: $(PHONY)

View file

@ -206,3 +206,97 @@ Engine settings
A few more options are possible, but they are pretty specific to some A few more options are possible, but they are pretty specific to some
engines, and so won't be described here. engines, and so won't be described here.
.. _settings location:
settings.yml location
=====================
First, searx will try to load settings.yml from these locations:
1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable.
2. ``/etc/searx/settings.yml``
If these files don't exist (or are empty or can't be read), searx uses the :origin:`searx/settings.yml` file.
.. _ settings use_default_settings:
use_default_settings
====================
.. note::
If searx is cloned from a git repository, most probably there is no need to have an user settings.
The user defined settings.yml can relied on the default configuration :origin:`searx/settings.yml` using ``use_default_settings: True``.
In the following example, the actual settings are the default settings defined in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and the ``bind_address``:
.. code-block:: yaml
use_default_settings: true
server:
secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
server:
bind_address: "0.0.0.0"
With ``use_default_settings: True``, each settings can be override in a similar way with one exception, the ``engines`` section:
* If the ``engines`` section is not defined in the user settings, searx uses the engines from the default setttings (the above example).
* If the ``engines`` section is defined then:
* searx loads only the engines declare in the user setttings.
* searx merges the configuration according to the engine name.
In the following example, only three engines are available. Each engine configuration is merged with the default configuration.
.. code-block:: yaml
use_default_settings: true
server:
secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
engines:
- name: wikipedia
- name: wikidata
- name: ddg definitions
Another example where four engines are available. The arch linux wiki engine has a :ref:`token<private engines>`.
.. code-block:: yaml
use_default_settings: true
server:
secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
- name: wikipedia
- name: wikidata
- name: ddg definitions
automatic update
----------------
The following comand creates or updates a minimal user settings (a secret key is defined if it is not already the case):
.. code-block:: sh
make SEARX_SETTINGS_PATH=/etc/searx/settings.yml user-settings.update
Set ``SEARX_SETTINGS_PATH`` to your user settings path.
As soon the user settings contains an ``engines`` section, it becomes difficult to keep the engine list updated.
The following command creates or updates the user settings including the ``engines`` section:
.. code-block:: sh
make SEARX_SETTINGS_PATH=/etc/searx/settings.yml user-settings.update.engines
After that ``/etc/searx/settings.yml``
* has a ``secret key``
* has a ``engine`` section if it is not already the case, moreover the command:
* has deleted engines that do not exist in the default settings.
* has added engines that exist in the default settings but are not declare in the user settings.

View file

@ -7,6 +7,8 @@ enabled engines on their instances. It might be because they do not want to
expose some private information through an offline engine. Or they expose some private information through an offline engine. Or they
would rather share engines only with their trusted friends or colleagues. would rather share engines only with their trusted friends or colleagues.
.. _private engines:
Private engines Private engines
=============== ===============

View file

@ -16,39 +16,15 @@ along with searx. If not, see < http://www.gnu.org/licenses/ >.
''' '''
import logging import logging
import searx.settings
from os import environ from os import environ
from os.path import realpath, dirname, join, abspath, isfile from os.path import realpath, dirname, join, abspath, isfile
from io import open
from yaml import safe_load
searx_dir = abspath(dirname(__file__)) searx_dir = abspath(dirname(__file__))
engine_dir = dirname(realpath(__file__)) engine_dir = dirname(realpath(__file__))
static_path = abspath(join(dirname(__file__), 'static')) static_path = abspath(join(dirname(__file__), 'static'))
settings, settings_load_message = searx.settings.load_settings()
def check_settings_yml(file_name):
if isfile(file_name):
return file_name
else:
return None
# find location of settings.yml
if 'SEARX_SETTINGS_PATH' in environ:
# if possible set path to settings using the
# enviroment variable SEARX_SETTINGS_PATH
settings_path = check_settings_yml(environ['SEARX_SETTINGS_PATH'])
else:
# if not, get it from searx code base or last solution from /etc/searx
settings_path = check_settings_yml(join(searx_dir, 'settings.yml')) or check_settings_yml('/etc/searx/settings.yml')
if not settings_path:
raise Exception('settings.yml not found')
# load settings
with open(settings_path, 'r', encoding='utf-8') as settings_yaml:
settings = safe_load(settings_yaml)
if settings['ui']['static_path']: if settings['ui']['static_path']:
static_path = settings['ui']['static_path'] static_path = settings['ui']['static_path']
@ -58,7 +34,6 @@ enable debug if
the environnement variable SEARX_DEBUG is 1 or true the environnement variable SEARX_DEBUG is 1 or true
(whatever the value in settings.yml) (whatever the value in settings.yml)
or general.debug=True in settings.yml or general.debug=True in settings.yml
disable debug if disable debug if
the environnement variable SEARX_DEBUG is 0 or false the environnement variable SEARX_DEBUG is 0 or false
(whatever the value in settings.yml) (whatever the value in settings.yml)
@ -78,7 +53,7 @@ else:
logging.basicConfig(level=logging.WARNING) logging.basicConfig(level=logging.WARNING)
logger = logging.getLogger('searx') logger = logging.getLogger('searx')
logger.debug('read configuration from %s', settings_path) logger.info(settings_load_message)
logger.info('Initialisation done') logger.info('Initialisation done')
if 'SEARX_SECRET' in environ: if 'SEARX_SECRET' in environ:

View file

@ -31,3 +31,11 @@ class SearxParameterException(SearxException):
self.message = message self.message = message
self.parameter_name = name self.parameter_name = name
self.parameter_value = value self.parameter_value = value
class SearxSettingsException(SearxException):
def __init__(self, message, filename):
super().__init__(message)
self.message = message
self.filename = filename

91
searx/settings.py Normal file
View file

@ -0,0 +1,91 @@
import collections.abc
import yaml
from searx.exceptions import SearxSettingsException
from os import environ
from os.path import dirname, join, abspath, isfile
searx_dir = abspath(dirname(__file__))
def check_settings_yml(file_name):
if isfile(file_name):
return file_name
else:
return None
def load_yaml(file_name):
try:
with open(file_name, 'r', encoding='utf-8') as settings_yaml:
settings = yaml.safe_load(settings_yaml)
if not isinstance(settings, dict) or len(settings) == 0:
raise SearxSettingsException('Empty file', file_name)
return settings
except IOError as e:
raise SearxSettingsException(e, file_name)
except yaml.YAMLError as e:
raise SearxSettingsException(e, file_name)
def get_default_settings_path():
return check_settings_yml(join(searx_dir, 'settings.yml'))
def get_user_settings_path():
# find location of settings.yml
if 'SEARX_SETTINGS_PATH' in environ:
# if possible set path to settings using the
# enviroment variable SEARX_SETTINGS_PATH
return check_settings_yml(environ['SEARX_SETTINGS_PATH'])
else:
# if not, get it from searx code base or last solution from /etc/searx
return check_settings_yml('/etc/searx/settings.yml')
def update_dict(d, u):
for k, v in u.items():
if isinstance(v, collections.abc.Mapping):
d[k] = update_dict(d.get(k, {}), v)
else:
d[k] = v
return d
def update_settings(default_settings, user_settings):
for k, v in user_settings.items():
if k == 'use_default_settings':
continue
elif k == 'engines':
default_engines = default_settings[k]
default_engines_dict = dict((definition['name'], definition) for definition in default_engines)
default_settings[k] = [update_dict(default_engines_dict[definition['name']], definition)
for definition in v]
else:
update_dict(default_settings[k], v)
return default_settings
def load_settings(load_user_setttings=True):
default_settings_path = get_default_settings_path()
user_settings_path = get_user_settings_path()
if user_settings_path is None or not load_user_setttings:
# no user settings
return (load_yaml(default_settings_path),
'load the default settings from {}'.format(default_settings_path))
# user settings
user_settings = load_yaml(user_settings_path)
if user_settings.get('use_default_settings'):
# the user settings are merged with the default configuration
default_settings = load_yaml(default_settings_path)
update_settings(default_settings, user_settings)
return (default_settings,
'merge the default settings ( {} ) and the user setttings ( {} )'
.format(default_settings_path, user_settings_path))
# the user settings, fully replace the default configuration
return (user_settings,
'load the user settings from {}'.format(user_settings_path))

View file

@ -0,0 +1,98 @@
#!/usr/bin/env python
# set path
from sys import path
from os.path import realpath, dirname, join
path.append(realpath(dirname(realpath(__file__)) + '/../'))
import argparse
import sys
import string
import ruamel.yaml
import secrets
import collections
from ruamel.yaml.scalarstring import SingleQuotedScalarString, DoubleQuotedScalarString
from searx.settings import load_settings, check_settings_yml, get_default_settings_path
from searx.exceptions import SearxSettingsException
RANDOM_STRING_LETTERS = string.ascii_lowercase + string.digits + string.ascii_uppercase
def get_random_string():
r = [secrets.choice(RANDOM_STRING_LETTERS) for _ in range(64)]
return ''.join(r)
def main(prog_arg):
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
yaml.indent(mapping=4, sequence=1, offset=2)
user_settings_path = prog_args.get('user-settings-yaml')
try:
default_settings, _ = load_settings(False)
if check_settings_yml(user_settings_path):
with open(user_settings_path, 'r', encoding='utf-8') as f:
user_settings = yaml.load(f.read())
new_user_settings = False
else:
user_settings = yaml.load('use_default_settings: True')
new_user_settings = True
except SearxSettingsException as e:
sys.stderr.write(str(e))
return
if not new_user_settings and not user_settings.get('use_default_settings'):
sys.stderr.write('settings.yml already exists and use_default_settings is not True')
return
user_settings['use_default_settings'] = True
use_default_settings_comment = "settings based on " + get_default_settings_path()
user_settings.yaml_add_eol_comment(use_default_settings_comment, 'use_default_settings')
if user_settings.get('server', {}).get('secret_key') in [None, 'ultrasecretkey']:
user_settings.setdefault('server', {})['secret_key'] = DoubleQuotedScalarString(get_random_string())
user_engines = user_settings.get('engines')
if user_engines:
has_user_engines = True
user_engines_dict = dict((definition['name'], definition) for definition in user_engines)
else:
has_user_engines = False
user_engines_dict = {}
user_engines = []
# remove old engines
if prog_arg.get('add-engines') or has_user_engines:
default_engines_dict = dict((definition['name'], definition) for definition in default_settings['engines'])
for i, engine in enumerate(user_engines):
if engine['name'] not in default_engines_dict:
del user_engines[i]
# add new engines
if prog_arg.get('add-engines'):
for engine in default_settings.get('engines', {}):
if engine['name'] not in user_engines_dict:
user_engines.append({'name': engine['name']})
user_settings['engines'] = user_engines
# output
if prog_arg.get('dry-run'):
yaml.dump(user_settings, sys.stdout)
else:
with open(user_settings_path, 'w', encoding='utf-8') as f:
yaml.dump(user_settings, f)
def parse_args():
parser = argparse.ArgumentParser(description='Update user settings.yml')
parser.add_argument('--add-engines', dest='add-engines', default=False, action='store_true', help='Add new engines')
parser.add_argument('--dry-run', dest='dry-run', default=False, action='store_true', help='Dry run')
parser.add_argument('user-settings-yaml', type=str)
return vars(parser.parse_args())
if __name__ == '__main__':
prog_args = parse_args()
main(prog_args)