Improve installer tasks and installer documentation.

docs/installers.rst

@@ -2,6 +2,8 @@
 Writing installers
 ==================
 
+See an example installer at the end of this document.
+
 Fetching required files
 =======================
 
@@ -51,7 +53,7 @@ directly but make the installer extract it from an archive or something, you
 can reference the rom with the ``main_file`` parameter.
 Example: ``main_file: game.rom``
 
-For browser games, specify the game's URL with ``main_file``.
+For web games, specify the game's URL (or filename) with ``main_file``.
 Example: ``main_file: http://www...``
 
 Presetting game parameters
@@ -63,19 +65,50 @@ parameters depend on the runner:
 *   linux: ``args`` (optional command arguments), ``working_dir``
     (optional working directory, defaults to the exe's dir).
 
-*   wine:  ``args``, ``prefix`` (optional Wine prefix), ``working_dir`` (optional
+*   wine:  ``args``, ``arch`` (optional WINEARCH), ``prefix`` (optional Wine prefix), ``working_dir`` (optional
     working directory, defaults to the exe's dir).
 
 *   winesteam: ``args``, ``prefix`` (optional Wine prefix).
 
-Example:
+Example (Windows game):
 
 ::
 
     game:
-        exe: drive_c/Game/game.exe
+      exe: drive_c/Game/game.exe
-        prefix: $GAMEDIR
+      prefix: $GAMEDIR
-        args: -arg
+      args: -arg
+
+Runner configuration
+--------------------
+
+The runner can be preconfigured from the installer.
+The name of the directive is the slug name of the runner,
+for example ``wine``. Available parameters depend on the runner.
+The best way to set this is to add the game to Lutris, tweak the
+runner config and then copy it from ``.config/lutris/games/<game name and id>.yml``.
+
+Example for Wine (set wine version for this installer):
+
+::
+
+    wine:
+      version: overwatch-2.15-x86_64
+
+System configuration
+--------------------
+
+The ``system`` directive lets you preset the system config for the game.
+
+Example (setting some environment variables):
+
+::
+
+    system:
+      env:
+        __GL_SHADER_DISK_CACHE: '1'
+        __GL_THREADED_OPTIMIZATIONS: '1'
+        mesa_glthread: 'true'
 
 Mods and add-ons
 ----------------
@@ -207,7 +240,7 @@ reference a ``file id`` or a path, ``args`` to add command arguments,
 to set the directory to execute the command in (defaults to the install path).
 The command is executed within the Lutris Runtime (resolving most shared
 library dependencies). The file is made executable if necessary, no need to run
-chmodx before.
+chmodx before. You can also use ``env`` (environment variables), ``exclude_processes`` (space-separated list of processes to exclude from being watched), ``include_processes`` (the opposite of ``exclude_processes``, is used to override Lutris' built-in exclude list) and ``disable_runtime`` (run a process without the Lutris Runtime, useful for running system binaries).
 
 Example:
 
@@ -217,14 +250,42 @@ Example:
         args: --argh
         file: $great-id
         terminal: true
+        env:
+          key: value
+
+You can use the ``command`` parameter instead of ``file`` and ``args``. This
+lets you run bash/shell commands easier. ``bash`` is used and is added to ``include_processes`` internally.
+
+Example:
+
+::
+
+    - execute:
+        command: 'echo Hello World! | cat'
+
+Writing files
+-------------
+
+Writing text files
+~~~~~~~~~~~~~~~~~~
+
+Create or overwrite a file with the ``write_file`` directive. Use the ``file`` (an absolute path or a ``file id``) and ``content`` parameters.
+
+Example:
+
+::
+
+    - write_file:
+        file: $GAMEDIR/myfile.txt
+        content: 'This is the contents of the file.'
 
 Writing into an INI type config file
-------------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Modify or create a config file with the ``write_config`` directive. A config file
 is a text file composed of key=value (or key: value) lines grouped under
 [sections]. Use the ``file`` (an absolute path or a ``file id``), ``section``,
-``key`` and ``value`` parameters. Not that the file is entirely rewritten and
+``key`` and ``value`` parameters. Note that the file is entirely rewritten and
 comments are left out; Make sure to compare the initial and resulting file
 to spot any potential parsing issues.
 
@@ -233,11 +294,37 @@ Example:
 ::
 
     - write_config:
-        file: $GAMEDIR/game.ini
+        file: $GAMEDIR/myfile.ini
         section: Engine
         key: Renderer
         value: OpenGL
 
+Writing into a JSON type file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Modify or create a JSON file with the ``write_json`` directive.
+Use the ``file`` (an absolute path or a ``file id``) and ``data`` parameters.
+Note that the file is entirely rewritten; Make sure to compare the initial
+and resulting file to spot any potential parsing issues. You can set the optional parameter ``merge`` to ``false`` if you want to overwrite the JSON file instead of updating it.
+
+Example:
+
+::
+
+    - write_json:
+        file: $GAMEDIR/myfile.json
+        data:
+          Sound:
+            Enabled: 'false'
+
+This writes (or updates) a file with the following content:
+::
+
+    {
+      "Sound": {
+        "Enabled": "false"
+      }
+    }
 
 Running a task provided by a runner
 -----------------------------------
@@ -269,7 +356,9 @@ Currently, the following tasks are implemented:
 *   wine / winesteam: ``wineexec`` Runs a windows executable. Parameters are
     ``executable`` (``file ID`` or path), ``args`` (optional arguments passed
     to the executable), ``prefix`` (optional WINEPREFIX),
-    ``working_dir`` (optional working directory).
+    ``arch`` (optional WINEARCH), ``working_dir`` (optional working directory),
+    ``exclude_processes`` (optional space-separated list of processes to exclude from
+    being watched), ``env`` (optional environment variables), ``overrides`` (optional dll overrides).
 
     Example:
 
@@ -282,7 +371,7 @@ Currently, the following tasks are implemented:
             args: --windowed
 
 *   wine / winesteam: ``winetricks`` Runs winetricks with the ``app`` argument.
-    ``prefix`` is an optional WINEPREFIX path.
+    ``prefix`` is an optional WINEPREFIX path. You can run many tricks at once by adding more to the ``app`` parameter (space-separated).
 
     By default Winetricks will run in silent mode but that can cause issues
     with some components such as XNA. In such cases, you can provide the
@@ -379,10 +468,49 @@ Trying the installer locally
 ============================
 
 If needed (i.e. you didn't download the installer first from the website), add
-the ``runner`` and ``name`` directives. The value for ``runner`` must be the
+the ``name``, ``game_slug``, ``slug`` and ``runner`` directives. The value for
-slug name for the runner. (E.g. winesteam for Steam Windows.)
+``runner`` must be the slug name for the runner. (E.g. winesteam for Steam Windows.)
-Save your script in a file and use the following command in a terminal:
+You can also add ``version``, ``description`` and ``notes`` to the installer file.
-``lutris -i /path/to/file``
+Under ``script``, add ``files``, ``installer``, ``game`` and other installer
+directives. See below for an example.
+Save your script in a .yaml file and use the following command in a terminal:
+``lutris -i /path/to/file.yaml``
+
+Example Linux game:
+
+::
+
+    name: My Game
+    game_slug: my-game
+    version: Installer
+    slug: my-game-installer
+    runner: linux
+
+    script:
+      game:
+        exe: $GAMEDIR/mygame
+        args: --some-arg
+
+      files:
+      - myfile: http://example.com/mygame.zip
+
+      installer:
+      - chmodx: $GAMEDIR/mygame
+
+When submitting the installer script to lutris.net, only copy the script part. Remove the two space indentation:
+
+::
+
+    game:
+      exe: $GAMEDIR/mygame
+      args: --some-arg
+
+    files:
+    - myfile: http://example.com
+
+    installer:
+    - chmodx: $GAMEDIR/mygame
+
 
 
 Calling the online installer

lutris/installer/commands.py

@@ -1,7 +1,9 @@
 import multiprocessing
 import os
+import stat
 import shutil
 import shlex
+import json
 
 from gi.repository import GLib
 
@@ -16,6 +18,19 @@ from lutris.runners import wine, import_task
 from lutris.thread import LutrisThread
 
 
+def selective_merge(base_obj, delta_obj):
+    """ used by write_json """
+    if not isinstance(base_obj, dict):
+        return delta_obj
+    common_keys = set(base_obj).intersection(delta_obj)
+    new_keys = set(delta_obj).difference(common_keys)
+    for k in common_keys:
+        base_obj[k] = selective_merge(base_obj[k], delta_obj[k])
+    for k in new_keys:
+        base_obj[k] = delta_obj[k]
+    return base_obj
+
+
 class CommandsMixin(object):
     """The directives for the `installer:` part of the install script."""
 
@@ -49,32 +64,59 @@ class CommandsMixin(object):
 
     def chmodx(self, filename):
         filename = self._substitute(filename)
-        os.popen('chmod +x "%s"' % filename)
+        st = os.stat(filename)
+        os.chmod(filename, st.st_mode | stat.S_IEXEC)
 
     def execute(self, data):
         """Run an executable file."""
         args = []
         terminal = None
         working_dir = None
+        env = {}
         if isinstance(data, dict):
-            self._check_required_params('file', data, 'execute')
+            if 'command' not in data and 'file' not in data:
-            file_ref = data['file']
+                raise ScriptingError('Parameter file or command is mandatory '
+                                     'for the execute command', data)
+            elif 'command' in data and 'file' in data:
+                raise ScriptingError('Parameters file and command can\'t be '
+                                     'used at the same time for the execute '
+                                     'command', data)
+            file_ref = data.get('file', '')
+            command = data.get('command', '')
             args_string = data.get('args', '')
             for arg in shlex.split(args_string):
                 args.append(self._substitute(arg))
             terminal = data.get('terminal')
             working_dir = data.get('working_dir')
+            if not data.get('disable_runtime', False):
+                env.update(runtime.get_env())
+            userenv = data.get('env', {})
+            for key in userenv:
+                v = userenv[key]
+                userenv[key] = self._get_file(v) or self._substitute(v)
+            env.update(userenv)
+            include_processes = data.get('include_processes', '').split(' ')
+            exclude_processes = data.get('exclude_processes', '').split(' ')
+        elif isinstance(data, str):
+            command = data
+            include_processes = []
+            exclude_processes = []
         else:
-            file_ref = data
+            raise ScriptingError('No parameters supplied to execute command.', data)
 
-        # Determine whether 'file' value is a file id or a path
+        if command:
-        exec_path = self._get_file(file_ref) or self._substitute(file_ref)
+            command = command.strip()
+            command = self._get_file(command) or self._substitute(command)
+            file_ref = 'bash'
+            args = ['-c', command]
+            include_processes.append('bash')
+        else:
+            # Determine whether 'file' value is a file id or a path
+            file_ref = self._get_file(file_ref) or self._substitute(file_ref)
+
+        exec_path = system.find_executable(file_ref)
         if not exec_path:
-            raise ScriptingError("Unable to find file %s" % file_ref,
+            raise ScriptingError("Unable to find executable %s" % file_ref)
-                                 file_ref)
-        if not os.path.exists(exec_path):
-            raise ScriptingError("Unable to find required executable",
-                                 exec_path)
         if not os.access(exec_path, os.X_OK):
             self.chmodx(exec_path)
 
@@ -86,8 +128,9 @@ class CommandsMixin(object):
 
         command = [exec_path] + args
         logger.debug("Executing %s" % command)
-        thread = LutrisThread(command, env=runtime.get_env(), term=terminal,
+        thread = LutrisThread(command, env=env, term=terminal, cwd=working_dir,
-                              cwd=self.target_path)
+                              include_processes=include_processes,
+                              exclude_processes=exclude_processes)
         thread.start()
         GLib.idle_add(self.parent.attach_logger, thread)
         self.heartbeat = GLib.timeout_add(1000, self._monitor_task, thread)
@@ -340,16 +383,61 @@ class CommandsMixin(object):
             return False
         return True
 
+    def write_file(self, params):
+        """Write text to a file."""
+        self._check_required_params(['file', 'content'], params, 'write_file')
+
+        # Get file
+        file = self._get_file(params['file']) or self._substitute(params['file'])
+
+        # Create dir if necessary
+        basedir = os.path.dirname(file)
+        if not os.path.exists(basedir):
+            os.makedirs(basedir)
+
+        mode = params.get('mode', 'w')
+
+        with open(file, mode) as f:
+            f.write(params['file'])
+
+    def write_json(self, params):
+        """Write data into a json file."""
+        self._check_required_params(['file', 'data'], params, 'write_json')
+
+        # Get file
+        file = self._get_file(params['file']) or self._substitute(params['file'])
+
+        # Create dir if necessary
+        basedir = os.path.dirname(file)
+        if not os.path.exists(basedir):
+            os.makedirs(basedir)
+
+        merge = params.get('merge', True)
+
+        with open(file, 'a+') as f:
+            pass
+
+        with open(file, 'r+' if merge else 'w') as f:
+            data = {}
+            if merge:
+                try:
+                    data = json.load(f)
+                except ValueError:
+                    pass
+
+            data = selective_merge(data, params.get('data', {}))
+            f.seek(0)
+            f.write(json.dumps(data, indent=2))
+
     def write_config(self, params):
         """Write a key-value pair into an INI type config file."""
         self._check_required_params(['file', 'section', 'key', 'value'],
                                     params, 'write_config')
         # Get file
-        config_file = self._get_file(params['file'])
+        config_file = (self._get_file(params['file']) or
-        if not config_file:
+                       self._substitute(params['file']))
-            config_file = self._substitute(params['file'])
 
-        # Create it if necessary
+        # Create dir if necessary
         basedir = os.path.dirname(config_file)
         if not os.path.exists(basedir):
             os.makedirs(basedir)

lutris/runners/wine.py

@@ -135,7 +135,8 @@ def create_prefix(prefix, wine_path=None, arch='win32'):
 
 def wineexec(executable, args="", wine_path=None, prefix=None, arch=None,
              working_dir=None, winetricks_wine='', blocking=False,
-             config=None, include_processes=[]):
+             config=None, include_processes=[], exclude_processes=[],
+             disable_runtime=False, env={}, overrides=None):
     """
     Execute a Wine command.
 
@@ -174,34 +175,42 @@ def wineexec(executable, args="", wine_path=None, prefix=None, arch=None,
         wine_bin = winetricks_wine if winetricks_wine else wine_path
         create_prefix(prefix, wine_path=wine_bin, arch=arch)
 
-    env = {
+    wineenv = {
         'WINEARCH': arch
     }
     if winetricks_wine:
-        env['WINE'] = winetricks_wine
+        wineenv['WINE'] = winetricks_wine
     else:
-        env['WINE'] = wine_path
+        wineenv['WINE'] = wine_path
     if prefix:
-        env['WINEPREFIX'] = prefix
+        wineenv['WINEPREFIX'] = prefix
 
     wine_config = config or LutrisConfig(runner_slug='wine')
-    if not wine_config.system_config['disable_runtime'] and not runtime.is_disabled():
+    if (not wine_config.system_config['disable_runtime'] and
-        env['LD_LIBRARY_PATH'] = ':'.join(runtime.get_paths())
+            not runtime.is_disabled() and not disable_runtime):
+        wineenv['LD_LIBRARY_PATH'] = ':'.join(runtime.get_paths())
+
+    if overrides:
+        wineenv['WINEDLLOVERRIDES'] = get_overrides_env(overrides)
+
+    wineenv.update(env)
 
     command = [wine_path]
     if executable:
         command.append(executable)
     command += shlex.split(args)
     if blocking:
-        return system.execute(command, env=env, cwd=working_dir)
+        return system.execute(command, env=wineenv, cwd=working_dir)
     else:
-        thread = LutrisThread(command, runner=wine(), env=env, cwd=working_dir,
+        thread = LutrisThread(command, runner=wine(), env=wineenv, cwd=working_dir,
-                              include_processes=include_processes)
+                              include_processes=include_processes,
+                              exclude_processes=exclude_processes)
         thread.start()
         return thread
 
 
-def winetricks(app, prefix=None, arch=None, silent=True, wine_path=None, config=None):
+def winetricks(app, prefix=None, arch=None, silent=True,
+               wine_path=None, config=None, disable_runtime=False):
     """Execute winetricks."""
     winetricks_path = os.path.join(datapath.get(), 'bin/winetricks')
     if arch not in ('win32', 'win64'):
@@ -214,7 +223,8 @@ def winetricks(app, prefix=None, arch=None, silent=True, wine_path=None, config=
     if str(silent).lower() in ('yes', 'on', 'true'):
         args = "--unattended " + args
     return wineexec(None, prefix=prefix, winetricks_wine=winetricks_wine,
-                    wine_path=winetricks_path, arch=arch, args=args, config=config)
+                    wine_path=winetricks_path, arch=arch, args=args,
+                    config=config, disable_runtime=disable_runtime)
 
 
 def winecfg(wine_path=None, prefix=None, arch='win32', config=None):
@@ -760,7 +770,7 @@ class wine(Runner):
         wineexec("regedit", wine_path=self.get_executable(), prefix=self.prefix_path, config=self)
 
     def run_winetricks(self, *args):
-        winetricks('', prefix=self.prefix_path, wine_path=self.get_executable(), config=self)
+        winetricks('', prefix=self.prefix_path, wine_path=self.get_executable(), config=self, disable_runtime=True)
 
     def run_joycpl(self, *args):
         joycpl(prefix=self.prefix_path, wine_path=self.get_executable(), config=self)

lutris/thread.py

@@ -28,7 +28,7 @@ EXCLUDED_PROCESSES = (
     'lutris', 'python', 'python3',
     'bash', 'sh', 'tee', 'tr', 'zenity', 'xkbcomp', 'xboxdrv',
     'steam', 'Steam.exe', 'steamer', 'steamerrorrepor', 'gameoverlayui',
-    'SteamService.ex', 'steamwebhelper', 'steamwebhelper.', 'PnkBstrA.exe',
+    'SteamService.exe', 'steamwebhelper', 'steamwebhelper.', 'PnkBstrA.exe',
     'control', 'winecfg.exe', 'wdfmgr.exe', 'wineconsole', 'winedbg',
 )
 
@@ -38,7 +38,7 @@ class LutrisThread(threading.Thread):
     debug_output = True
 
     def __init__(self, command, runner=None, env={}, rootpid=None, term=None,
-                 watch=True, cwd=None, include_processes=[], log_buffer=None):
+                 watch=True, cwd=None, include_processes=[], exclude_processes=[], log_buffer=None):
         """Thread init"""
         threading.Thread.__init__(self)
         self.env = env
@@ -59,6 +59,7 @@ class LutrisThread(threading.Thread):
         self.daemon = True
         self.error = None
         self.include_processes = include_processes
+        self.exclude_processes = exclude_processes
         self.log_buffer = log_buffer
         self.stdout_monitor = None
         self.monitored_processes = None  # Keep a copy of the monitored processes to allow comparisons
@@ -246,7 +247,11 @@ class LutrisThread(threading.Thread):
             if child.pid in self.old_pids:
                 processes['external'].append(str(child))
                 continue
-            if child.name in EXCLUDED_PROCESSES and child.name not in self.include_processes:
+
+            if (child.name and
+                (child.name in EXCLUDED_PROCESSES or
+                 child.name in self.exclude_processes) and
+               child.name not in self.include_processes):
                 processes['excluded'].append(str(child))
                 continue
             num_watched_children += 1

lutris/util/system.py

@@ -114,7 +114,7 @@ def get_md5_hash(filename):
 
 
 def find_executable(exec_name, quiet=False):
-    if not exec_name:
+    if not exec_name and not quiet:
         raise ValueError("find_executable: exec_name required")
     return shutil.which(exec_name)