Installing Streams Quick Start Edition with Docker

Edit me

IBM® Streams Quick Start Edition can help you get started with Streams quickly, without having to install a Streams cluster environment.

Introduction

This document describes the installation, configuration, first steps, and common Docker management scenarios for Streams Quick Start Edition running in a Docker environment.

Overview

Here’s an overview of the steps to get up and running:

Supported environments

The following operating system and Docker environments are supported for Streams Quick Start Edition 4.3.

Operating system Docker
Windows 10 Docker Community Edition 17.03.1-ce or later
MacOS El Capitan 10.11 or later Docker Community Edition 17.03.1-ce or later
Linux systemd-based distribution Docker Community Edition 17.03.1-ce or later.

Note: If you use Standard Docker (Red Hat), you must adjust the Docker environment manually to allow for 50 GB of Docker storage.

Docker configuration requirements

Configure your Docker environment as follows.

Setting Minimum Recommended
CPUs 2 4
Memory 4 GB 8 GB
Disk space 20 GB 50 GB or greater depending upon number and size of projects.

Installing and configuring Docker

  1. Download and install Docker-CE 17.03.1 or later from:

    https://store.docker.com/editions/community/docker-ce-desktop-windows

    Note: Installing Docker might conflict with settings required for other VM technologies such as Oracle VM VirtualBox. The installation might also require that you set your VM technology settings in your BIOS settings. If this is necessary, Docker warns you when you try to start it and tells you which settings to change.

  2. After you install Docker, configure it as follows:
    1. Right-click the Docker icon in the system tray and select Settings.
    2. Under Setting, select Shared Drives. Share your hard disk drive (usually your C: drive).
    3. Under Settings, select Advanced and configure the CPUs (minimum 2, recommended 4) and memory (minimum 4 GB, recommended 8 GB).
  3. Open a PowerShell session from the Windows menu and test that Docker is set up correctly by running this command:
        docker run hello-world
    

    If Docker is set up correctly, you will receive a response that includes the following text:

    Hello from Docker!
    This message shows that your installation appears to be working correctly.
    

  1. Download and install Docker Community Edition for Mac from:

    https://store.docker.com/editions/community/docker-ce-desktop-mac

  2. After you install Docker, configure it as follows:
    1. From the Docker icon in the menu, select Preferences > Advanced.
    2. Configure the CPUs (minimum 2, recommended 4) and memory (minimum 4 GB, recommended 8 GB).
  3. Open a Terminal session and test that Docker is set up correctly by running this command:
        docker run hello-world
    
    If Docker is set up correctly, you will receive a response that includes this text:
       Hello from Docker!
       This message shows that your installation appears to be working
       correctly.
        
  1. Use the OS package manager to install Docker Community Edition. (For Red Hat, install Standard Docker).
  2. If necessary, increase the Docker installation storage space for the Streams Quick Start Edition image.

    Tip: The amount of storage space that is needed depends on whether you map directories to the local file system and the amount and the size of your applications and data.

    • The Red Hat Standard Docker storage space is set to 10 GB. In this case, you must increase the storage space.
    • Storage space for Docker-ce is usually set to 20 GB. In this case, you might need to increase the storage space.

    1. To set the Docker default image size (Base Device Size), log in as root or use sudo to create or edit the /etc/docker/daemon.json file.
    2. Add or modify the following code, where XX represents the default image size in GB.
      • If you keep the /home/streamsadmin/workspace directory in the Docker image, set the storage space to at least 50 GB.
      • If you map the /home/streamsadmin/workspace to a local host directory, you can set the Docker storage to 20 GB, because projects and data can be stored on your local drive instead of inside the Docker container.
      {
        "storage-opts": ["dm.basesize=XXG"]
      }
      
  3. Restart the docker service:
    sudo systemctl restart docker
  4. Verify the default image size by entering the following command:
    docker info |grep "Base Device Size:"

Mapping Docker container directories to the local host file system

(Optional, but recommended)

During the Streams Quick Start Edition installation, you are prompted to map two Docker directories, /home/streamsadmin/workspace and /home/streamsadmin/hostdir, to local host directories.

Because the container has limited space, in most cases the best option is to map to external directories. Doing so will help prevent the container from exceeding its internal disk limits and make it easier for you to back up your project data and upgrade your Streams Docker container in the future.

  1. Create a directory on your host machine where you can keep both mapped directories isolated from other host machine files. For example, you might create a directory called mappedDockerFiles in the home directory of your host machine:

    mkdir <HOME DIRECTORY>/mappedDockerFiles

    Important: Do not map the Docker files directly to the top-level user home directory.

  2. Make note of the directory. You will provide it during the Quick Start Edition installation.

The installation creates the mapped directories in your local host file system within mappedDockerFiles for two directories, the internal workspace and hostdir subdirectories. During installation, you are prompted for the names that you want to use under ~/mappedDockerFiles.

The following explains the use of the two directories.

  • The workspace directory: Streams projects created using Streams Studio would be stored in /home/streamsadmin/workspace on the Docker container. By mapping the workspace directory to a folder on your local host, the projects stored in /home/streamsadmin/workspace will be stored on the mapped workspace directory on your local host machine. Storing them here instead of in the internal Docker container means that you can reuse this directory for easy recovery of your projects. For example, if you later upgrade your Streams4Docker installation, you use the Import function of Streams Studio to import the projects from a previous Streams4Docker installation.

  • The hostdir directory: If this directory is mapped to the local host file system, it corresponds to /home/streamsadmin/hostdir in the Docker container. Use it to store files that you want to share between your Docker container and your local host. Keep any large files that are used in your Streams4Docker container in this directory when possible, because files in this directory do not use up space inside the Docker container.

Download and install Streams Quick Start Edition

After you configure Docker and map your directories, you are ready to download and install Quick Start Edition.

You have two options:

Download and install from Docker Hub

The following sections contain sample commands for downloading and installing Quick Start Edition from Docker Hub.

The commands that you use to download and install the container on Windows depend on whether you choose to map the container directories.

Container with mapped directories

This command is based on using the following mapped directories, replacing user with your user name:

C:\user\Documents\DockerMapped\workspace
C:\user\Documents\DockerMapped\hostdir

Enter the following command, replacing user with your user name:

docker run --privileged -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v //c/user/Documents/DockerMapped/workspace:/home/streamsadmin/workspace -v //c/user/Documents/DockerMapped/hostdir:/home/streamsadmin/hostdir -p 8443:8443 -p 9975:9975 -p 8006-8016:8006-8016 -p 8444:8444 -p 8080:80 -p 5905:5901 -p 4022:22 --name streamsdocker4300 -h 'streamsqse.localdomain' ibmcom/streams-qse:4.3.0.0 INSTALL

Container without mapped directories

Enter the following command:

docker run --privileged -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 8443:8443 -p 9975:9975 -p 8006-8016:8006-8016 -p 8444:8444 -p 8080:80 -p 5905:5901 -p 4022:22 --name streamsdocker4300 -h 'streamsqse.localdomain' ibmcom/streams-qse:4.3.0.0 INSTALL

The commands that you use to download and install the container on Mac or Linux depend on whether you choose to map the container directories.

Container with mapped directories

This command is based on setting the MAPPED_WORKSPACE and MAPPED_HOSTDIR variables to the mapped directories you created, for example:

export MAPPED_WORKSPACE=$HOME/Documents/DockerMapped/workspace
export MAPPED_HOSTDIR=$HOME/Documents/DockerMapped/hostdir

After you export the mapped drives, enter the following command:

docker run --privileged -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v $MAPPED_WORKSPACE:/home/streamsadmin/workspace -v $MAPPED_HOSTDIR:/home/streamsadmin/hostdir -p 8443:8443 -p 9975:9975 -p 8006-8016:8006-8016 -p 8444:8444 -p 8080:80 -p 5905:5901 -p 4022:22 --name streamsdocker4300 -h 'streamsqse.localdomain' ibmcom/streams-qse:4.3.0.0 INSTALL

Container without mapped directories

Enter the following command:

docker run --privileged -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 8443:8443 -p 9975:9975 -p 8006-8016:8006-8016 -p 8444:8444 -p 8080:80 -p 5905:5901 -p 4022:22 --name streamsdocker4300 -h 'streamsqse.localdomain' ibmcom/streams-qse:4.3.0.0 INSTALL

Manual download and installation

Prerequisite: Make sure you are connected to the Internet.

  1. Click Download Streams Quick Start Edition to go to the IBM Streams Quick Start Edition page. Download IBM Streams Quick Start Edition V4.3.0 for Docker on Windows 10.
  2. Extract the Streams4WindowsDocker4300EL7.zip file into a directory.
  3. Open PowerShell and go to the Streams4WindowsDocker4300EL7 directory.
  4. Ensure that you have permissions to run a PowerShell script:
    1. In the PowerShell window, determine your PowerShell execution policy:
      get-executionpolicy
    2. If your permissions are restricted, change them to unrestricted:
      set-executionpolicy unrestricted

      Tip: After you complete the installation, optionally reset your execution policy back to restricted by entering the following command:

      set-executionpolicy restricted

  5. Run the install script by entering the following command:
    ./streamsdockerInstall.ps1
  6. Follow the installation instructions.
    1. Read and accept the license agreement page and the Notices page.
    2. Choose whether to map host local directories into the Docker container. If you have not created a local directory for this purpose, see Mapping Docker container directories to the local host file system.
    3. Confirm the directories and start the installation.
  1. Click Download Streams Quick Start Edition to go to the IBM Streams Quick Start Edition page and use the following instructions for your operating system.
  2. Extract the Streams4Docker4300EL7 file into a directory.
  3. From your Terminal session, change to the Streams4Docker4300EL7 directory.
  4. Run the install script by entering the following command:
    ./streamsdockerInstall.sh
  5. Follow the installation instructions.
    1. Read and accept the license agreement page and the Notices page.
    2. Choose whether to map host local directories into the Docker container. If you have not created a local directory for this purpose, see Mapping Docker container directories to the local host file system.
    3. Confirm the directories and start the installation.

The installation takes from 25 minutes to an hour or more depending on your host system configuration. When the installation is complete, you are returned to a Streams4Docker command prompt.

Configuring the hosts file

Streams operates by using a host name and expects DNS to provide the conversion from host name to IP address. Because Streams Quick Start Edition does not have a DNS server, you can simulate one by using the hosts file. Before you can access Streams Quick Start Edition, you must set the hosts file to point the host name streamsqse.localdomain to the 127.0.0.1 loopback address.

  1. Open a text editor as an administrator. For example, for Notepad, find Notepad in your Windows menu, right-click it, and select **Run as Administrator**.
  2. Open C:\Windows\System32\drivers\etc\hosts.

    Tip: Set the Notepad filename filter to "All Files" to see the `hosts` file.

  3. Append streamsqse streamsqse.localdomain to the line that has the loopback address. For example:
        127.0.0.1 localhost streamsqse streamsqse.localdomain
        
  4. Save and close the file.
  1. Open a text editor with root authority.
  2. Open /etc/hosts.
  3. Append streamsqse streamsqse.localdomain to the line that has the loopback address. For example:
        127.0.0.1 localhost streamsqse streamsqse.localdomain
        
  4. Save and close the file.

Accessing Streams Quick Start Edition with a VNC client

Use port 5905 to access Streams Quick Start Edition in the Docker container with a VNC client.

  1. Open your VNC Client and connect to the VNC server:

    streamsqse.localdomain:5905

  2. When you are prompted for a password, enter:

    passw0rd

The default user name for the Streams Console is streamsadmin.

On the Streams desktop, access the Streams applications from the Applications menu.

Accessing Streams Quick Start Edition with Secure Shell (SSH)

You can use ssh to access the container by specifying the streamsqse.localdomain and using port 4022. For example:

ssh -p 4022 streamsadmin@streamsqse.localdomain

By default, all passwords are passw0rd within an ssh session. If you use the user name streamsadmin and you want to run a command as root, you can use one of two methods:

  • sudo [command] Provide your streamsadmin password when you are prompted. You run commands by using sudo and return to streamsadmin ID after the command is completed.

  • sudo su - Provide your streamsadmin password when you are prompted. You will be logged in as root. To return to the streamsadmin user ID, enter: exit.

Managing the Docker container

The following commands are useful for managing your Docker container with Streams Quick Start Edition.

Opening a session on Docker

You can use the attach command or the exec command to open a session on Docker.

The difference between the attach and exec commands is that the attach command takes you into the main root session. The main root session which might be busy doing other functions (for example, it might be in the process of starting streams), which will prevent you from doing useful work, but will allow for better troubleshooting in case of a problem. The exec command opens a new separate root session.

Attaching to a running Docker container

  1. Open PowerShell (Windows) or Terminal (MacOS or Linux).

  2. Determine the name of the session by entering docker ps. The container streamsdocker4300 is displayed with a status of running.

  3. Attach to the container by entering this command:

    docker attach streamsdocker4300

    This command takes you to the Docker container Linux prompt.

Running a command in a running container with the exec command

Enter the Docker exec command:

docker exec -ti streamsdocker4300 /bin/bash

Detaching from the container

From within your attached Docker session, you can detach from the session and still leave session active.

  1. Use the appropriate key combination for your environment:
    • Windows or Linux: Ctrl-q-p
    • MacOS: command-q-p

    Important: If you are attached to a container and enter exit, it might cause the container to stop.

  2. (Optional) Close the PowerShell or Terminal session.

Pausing and restarting the container

You can pause a container when it is not in use. The container pauses and all activity inside the container stops. You can unpause the container to use it again.

  1. If you are running external programs that are connected to the container, close them.

  2. From a PowerShell (Windows) or Terminal (MacOS or Linux) session, enter the appropriate command:

    • To pause the container:
      docker pause streamsdocker4300
      
    • To unpause the container:
      docker unpause streamsdocker4300
      

Stopping and restarting the container

Typically, you manage a container by pausing and unpausing it. But sometimes you might need to stop and restart the container, for example, for a reboot.

From a PowerShell (Windows) or Terminal (MacOS or Linux) session, enter the appropriate command:

  • To stop a container:

    docker stop streamsdocker4300
    
  • To restart a container:

    docker restart streamsdocker4300
    

    Tip: When a container is restarted, it takes a few minutes for the system to come back up and start the domains. To see the status, you can use VNC to access the desktop, where you can see the yellow desktop icon (Streams Domain Starting) while the domain is starting. The icon turns green (Streams Domain Started) when the domain is ready. The icon eventually disappears.

Adjusting the desktop screen size for VNC

When you first access Streams Quick Start Edition with VNC, you will see the Streams desktop.

If you are accustomed to earlier versions of Streams Quick Start Edition, you will notice that there are no longer any desktop icons. All Streams Quick Start Edition resources have been moved to the Applications > Favorites menu, which you can access from the top left of the screen. There you will find the Streams applications and links to resource web pages.

To adjust the display size to closer match your physical display, follow these steps:

  1. Go to Applications > System Tools > Settings.

  2. Click Devices > Displays.

  3. In the Displays app, click Resolution.

  4. Select the resolution that best matches your physical display. Close the Resolution list by clicking ‘x’ in the corner. Click Apply.

  5. Click Keep Changes.

Known issues

Description: Firefox crashes often

Workaround:

  • Open Firefox and type about:config in the address bar. Accept the warning message.
  • Search for browser.tabs.remote.autostart and set it to false.
  • Repeat the above for the browser.tabs.remote.autostart.2, set it to false.
  • Restart Firefox.

Workaround #2:

If that still does not help, try installing and using Google Chrome.

wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
sudo yum localinstall google-chrome-stable_current_x86_64.rpm

Description: Running streamsdockerInstall.sh fails with the following message:

  TASK [install Streams prereq packages with yum] ********************************
  changed: [streamsqse.localdomain] => (item=readline)
  failed: [streamsqse.localdomain] (item=requests) => {“changed”: false, “cmd”: “/usr/bin/pip2 install -U requests”, “item”: “requests”, “msg”: “stdout: Collecting requests\n
  ...<snip>...
  Installing collected packages: certifi, chardet, idna, urllib3, requests\n  Found existing installation: chardet 2.2.1\n    Uninstalling chardet-2.2.1:\n      Successfully uninstalled chardet-2.2.1\n  Found existing installation: idna 2.4\n    Uninstalling idna-2.4:\n      Successfully uninstalled idna-2.4\n  Found existing installation: urllib3 1.10.2\n    Uninstalling urllib3-1.10.2:\n      Successfully uninstalled urllib3-1.10.2\n

  Found existing installation: requests 2.6.0\n\n:stderr: ipapython 4.5.0 requires pyldap>=2.4.15, which is not installed.\nrtslib-fb 2.1.63 has requirement pyudev>=0.16.1, but you’ll have pyudev 0.15 which is incompatible.\nipapython 4.5.0 has requirement dnspython>=1.15, but you’ll have dnspython 1.12.0 which is incompatible.\nCannot uninstall ‘requests’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\n”}

  changed: [streamsqse.localdomain] => (item=future)
  changed: [streamsqse.localdomain] => (item=dill)
      to retry, use: --limit @/root/ansible/streamsdockerCentos7.retry

  PLAY RECAP *********************************************************************
  streamsqse.localdomain     : ok=41   changed=32   unreachable=0    failed=1  

Cause: This might be caused by a change in the Python repositories.

Workaround:

  1. In the Quick Start Edition installation directory, locate the Ansible/streamsdockerCentos7.yaml file and open it in a text editor.
  2. Locate the following section:
     ######### Python Module installation ####################################
    
     name: install Streams prereq packages with yum
     pip: name= state=latest
     with_items:
    
     readline
     requests
     future
     dill
    
  3. Change requests to request.
  4. Save the file.
  5. Remove the old container:
     docker stop streamsdocker4300
     docker rm streamsdocker4300
    
  6. Reinstall Quick Start Edition.

Description: Streams Studio Project Explorer: Right-clicking a file or folder, and then selecting Show in > System Explorer throws a dbus error.

Cause: CentOS 6 Nautilus is not compatible with dbus API.

Workaround:

  1. Open Streams Studio.

  2. Go to Window > Preferences > General > Workspace.

  3. Set Command for launching system explorer as follows:

    nautilus "${selected_resource_parent_loc}"
    

Getting help

If your problem is not discussed in the known issues section, you can ask a question on the Streamsdev forum.