How to install a GUI for Amazon AWS EC2 or EMR with Amazon AMI

Here is my solution to run the GUI on Amazon AMI. I used this post as a starting point, but I had to make a lot of changes to make it work on Amazon AMI. I also added additional information to make this work automated enough so that a person who needs to repeatedly raise this environment can do it without much hassle.

Note: I include a lot of comments in this post. I apologize in advance, but I thought it might be useful for those who need to make changes if they can understand why various choices were made along the way.

, , . . 4 , .

1.

"-"

sudo yum groupinstall -y "Desktop"

Amazon AMI. Amazon AMI ( AAMI). AAMI Amazon, . epel, . epel Desktop, . Xfce ( ), . Xfce, Desktop. , , .

, , , centos Desktop. . . , , AAMI, .

Xfce Epel. epel AAMI, , Amazon. . epel, Amazon. , , Centos, . , - Centos, , Amazon Epel. , . . Xfce

installGui.sh

#!/bin/bash

# echo each command
set -x

# assumes RSRC_DIR and IS_EMR set by parent script
YUM_RSRC_DIR=$RSRC_DIR/yum

sudo yum -y update

# Most info I've found on installing a GUI on AWS suggests to install using
#> sudo yum groupinstall -y "Desktop"
# This group is not available by default on the Amazon Linux AMI.  The group
# is listed if the epel repo is enabled, but it is empty.  I tried installing
# the centos repo, which does have support for this group, but it simply end
# up having to many dependency version conflicts with packages already installed
# by the Amazon repos.
#
# I found the path of least resistance to be installing the group Xfce from
# the epel repo. The epel repo is already included in amazon image, just not enabled.
# So I'm guessing there was at least some consideration by Amazon to align
# the dependency versions of this repo with the Amazon repos.
#
# My general approach to this problem was to start with the last command:
#> sudo yum groupinstall -y Xfce
# which will generate a list of missing dependencies.  The script below
# essentially works backwards through that list to eliminate all the
# missing dependencies.
#
# In general, many of the dependencies required by Xfce are found in either
# the epel repo or the Amazon repos.  Most of the remaining dependencies can be
# found in the centos repo, and either don't have any further dependencies, or if they
# do those dependencies are satisfied with the centos repo with no collisions
# in the epel or amazon repo.  Then there are a couple of oddball dependencies
# to clean up.

# if yum-config-manager is not found then install yum-utils
#> sudo yum install yum-utils
sudo yum-config-manager --enable epel

# install centos repo
# place the repo config @  /etc/yum.repos.d/centos.repo
sudo cp $YUM_RSRC_DIR/yum.repos.d/centos.repo /etc/yum.repos.d/

# The config centos.repo specifies the key with a URL.  If for some reason the key
# must be in a local file, it can be found here: https://www.centos.org/keys/RPM-GPG-KEY-CentOS-6
# It can be installed to the right location in one step:
#> wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 https://www.centos.org/keys/RPM-GPG-KEY-CentOS-6
# Note, a key file must also be installed in the system key ring.  The docs are a bit confusing
# on this, I found that I needed to run both gpg AND then followed by rpm, eg:
#> sudo gpg --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
#> sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

# I found there are a lot of version conflicts between the centos, Amazon and epel repos.
# So I did not enable the centos repo generally.  Instead I used the --enablerepo switch
# enable it explicitly for each yum command that required it.  This only works for yum.  If
# rpm must be used, then yum-config-manager must be used to enable/disable repos as a
# separate step.
#
# Another problem I ran into was yum installing the 32-bit (*.i686) package rather than
# the 64-bit (*.x86_64) verision of the package.  I never figured out why.  So I had
# to specify the *.x86_64 package explicitly.  The search tools (eg. 'whatprovides')
# did not list the 64 bit package either even though a manual search through the
# package showed the 64 bit components were present.
#
# Sometimes it is difficult to determine which package must be in installed to satisfy
# a particular dependency.  'whatprovides' is a very useful tool for this
#> yum --enablerepo centos whatprovides libgdk_pixbuf-2.0.so.0
#> rpm -q --whatprovides libgdk_pixbuf

sudo yum --enablerepo centos install -y gdk-pixbuf2.x86_64
sudo yum --enablerepo centos install -y gtk2.x86_64
sudo yum --enablerepo centos install -y libnotify.x86_64
sudo yum --enablerepo centos install -y gnome-icon-theme
sudo yum --enablerepo centos install -y redhat-menus
sudo yum --enablerepo centos install -y gstreamer-plugins-base.x86_64

# problem when we get to libvte, installing libvte requires expat, which conflicts with amazon lib
# the centos package version was older and did not install right lib version
# but … the expat dependency was coming from a dependency on python-libs.
# the easiest workaround was to install python using the amazon repo, that in turn
# installs a version of python libs that is compatible with the version of libexpat on the system.

sudo yum install -y python
sudo yum --enablerepo centos install -y vte.x86_64

sudo yum --enablerepo centos install -y libical.x86_64
sudo yum --enablerepo centos install -y gnome-keyring.x86_64

# another sticky point, xfdesktop requires desktop-backgrounds-basic, but ‘whatprovides does not 
# provide any packages for this query (not sure why).  It turns out this is provided by the centos 
# repo, installing ‘desktop-backgrounds-basic will try to install the package redhat-logos, but 
# unfortunately this is obsoleted by Amazons generic-logos package
# The only way I could find to get around this was to erase the generic logos package.
# This doesn't seem too risky since this is just images for the desktop and menus.
#
sudo yum erase -y generic-logos

# Amazon repo must be disabled to prevent interference with the install
# of redhat-logos
sudo yum --disablerepo amzn-main --enablerepo centos install -y redhat-logos

# next problem is a dependency on dbus.  The dependency comes from dbus-x11 in 
# centos repo.  It requires dbus version 1.2.24, the amazon image already has
# version 1.6.12 installed.  Since the dbus-x11 is only used by the GUI package,
# easiest way around this is to install dbus-x11 with no dependency checks.
# So it will use the newer version of dbus (should be OK).  The main thing that could be a problem
# here is if it skips some other dependency.  When doing manually, its possible to run the install until
# the only error left is the dbus dependency.  Its a bit risky running in a script since, basically its assuming
# all the dependencies are already in place.
yumdownloader --enablerepo centos dbus-x11.x86_64
sudo rpm -ivh --nodeps dbus-x11-1.2.24-8.el6_6.x86_64.rpm
rm dbus-x11-1.2.24-8.el6_6.x86_64.rpm

sudo yum install -y xfdesktop.x86_64

# We need the version of poppler-glib from centos repo, but it is found in several repos.
# Disable the other repos for this step.
# On EMR systems a newer version of poppler is already installed.  So move up 1 level
# in dependency chain and force install of tumbler.

if [ $IS_EMR -eq 1 ]
then
    yumdownloader --enablerepo centos tumbler.x86_64
    sudo rpm -ivh --nodeps tumbler-0.1.21-1.el6.x86_64.rpm
else
    sudo yum --disablerepo amzn-main --disablerepo amzn-updates --disablerepo epel --enablerepo centos install -y poppler-glib
fi


sudo yum install  --enablerepo centos -y polkit-gnome.x86_64
sudo yum install  --enablerepo centos  -y control-center-filesystem.x86_64

sudo yum groupinstall -y Xfce

centos:

centos.repo

[centos]
name=CentOS mirror
baseurl=http://repo1.ash.innoscale.net/centos/6/os/x86_64/
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-6

Amazon AMI, . , VNC SSH , .

2. VNC

. , - 1 . , ec2, emr, root, ec2.

1. libXfont
2. tiger-vnc-
3. VNC
4. .vnc
5. xstartup .vnc
6. passwd .vnc
7. VNC

, :

, VNC SSH. , . , , SSH, - . , / VNC . , . vncservers. -localhost vnc . "-nolisten tcp" xserver . , [-SecurityTypes None " VNC , - ssh, .

xstartup , , VNC . , . Xfce, VNC. .

, , VNC , , passwd .vnc . . ssh vncpasswd. passwd .vnc, . , , VNC passwd. , , .

installGui.sh

#!/bin/bash

# echo each command
set -x

BIN_DIR="${BASH_SOURCE%/*}"
ROOT_DIR=$(dirname $BIN_DIR)
RSRC_DIR=$ROOT_DIR/rsrc
VNC_DIR=$RSRC_DIR/vnc

# Install user config files into ec2-user home directory
# if it is available.  In practice, this should always
# be true

if [ -d "/home/ec2-user" ]
then
   USER_ACCT=ec2-user
else
   USER_ACCT=hadoop
fi

HOME_DIR="/home"

# Use existence of hadoop home directory as proxy to determine if
# this is an EMR system.  Can be used later to differentiate
# steps on EC2 system vs EMR.
if [ -d "/home/hadoop" ]
then
    IS_EMR=1
else
    IS_EMR=0
fi


# execute Xfce desktop install
. "$BIN_DIR/installXfce.sh"

# now roughly follow the following from step 3: https://devopscube.com/setup-gui-for-amazon-ec2-linux/

sudo yum install -y pixman pixman-devel libXfont

sudo yum -y install tigervnc-server


# install the user account configuration file.
# This setup assumes the user will always connect to the VNC server
# through an SSH tunnel.  This is generally more secure, easier to
# configure and easier to get correct than trying to allow direct
# connections via TCP.
# Therefore, config VNC server to only accept local connections, and
# no password required.
sudo cp $VNC_DIR/vncservers-$USER_ACCT /etc/sysconfig/vncservers

# install the user account, vnc config files

sudo mkdir $HOME_DIR/$USER_ACCT/.vnc
sudo chown $USER_ACCT:$USER_ACCT $HOME_DIR/$USER_ACCT/.vnc

# need xstartup file to tell vncserver to start the window manager
sudo cp $VNC_DIR/xstartup $HOME_DIR/$USER_ACCT/.vnc/
sudo chown $USER_ACCT:$USER_ACCT $HOME_DIR/$USER_ACCT/.vnc/xstartup

# Even though the VNC server is config'd to not require a passwd, the
# server still looks for the passwd file when it starts the session.
# It will fail if the passwd file is not found.
# The first time these scripts are run, the final step will fail.
# Then manually run
#> vncpasswd
# It will create the file ~/.vnc/passwd.  Then save this file to persistent
# storage so that it can be installed to the user account during
# server initialization.

sudo cp $ROOT_DIR/home/user/.vnc/passwd $HOME_DIR/$USER_ACCT/.vnc/
sudo chown $USER_ACCT:$USER_ACCT $HOME_DIR/$USER_ACCT/.vnc/passwd

# This script will be running as root if called from the EC2 launch
# command.  VNC server needs to be started as the user that
# you will connect to the server as (eg. ec2-user, hadoop, etc.)
sudo su -c "sudo service vncserver start" -s /bin/sh $USER_ACCT

# how to stop vncserver
# vncserver -kill :1

# On the remote client
# 1. start the ssh tunner
#> ssh -i ~/.ssh/<YOUR_KEY_FILE>.pem -L 5901:localhost:5901 -N ec2-user@<YOUR_SERVER_PUBLIC_IP>
#    for debugging connection use -vvv switch
# 2. connect to the vnc server using client on the remote machine.  When
#    prompted for the IP address, use 'localhost:5901'
#    This connects to port 5901 on your local machine, which is where the ssh
#    tunnel is listening.

vncservers

# The VNCSERVERS variable is a list of display:user pairs.
#
# Uncomment the lines below to start a VNC server on display :2
# as my 'myusername' (adjust this to your own).  You will also
# need to set a VNC password; run 'man vncpasswd' to see how
# to do that.  
#
# DO NOT RUN THIS SERVICE if your local area network is
# untrusted!  For a secure way of using VNC, see this URL:
# http://kbase.redhat.com/faq/docs/DOC-7028

# Use "-nolisten tcp" to prevent X connections to your VNC server via TCP.

# Use "-localhost" to prevent remote VNC clients connecting except when
# doing so through a secure tunnel.  See the "-via" option in the
# 'man vncviewer' manual page.

# Use "-SecurityTypes None" to allow session login without a password.
# This should only be used in combination with "-localhost"
# Note: VNC server still looks for the passwd file in ~/.vnc directory
# when the session starts regardless of whether the user is
# required to enter a passwd.

# VNCSERVERS="2:myusername"
# VNCSERVERARGS[2]="-geometry 800x600 -nolisten tcp -localhost"
VNCSERVERS="1:ec2-user"
VNCSERVERARGS[1]="-geometry 1280x1024 -nolisten tcp -localhost -SecurityTypes None"

XStartup

#!/bin/sh

unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
# exec /etc/X11/xinit/xinitrc
/usr/share/vte/termcap/xterm &
/usr/bin/startxfce4 &

3.

, VNC- EC2, . SSH . 5901 - , VNC 1 vncservers. 2 5902 .. 5901 5901 .

ssh -i ~/.ssh/<YOUR_KEY_FILE>.pem -L 5901:localhost:5901 -N ec2-user@<YOUR_SERVER_PUBLIC_IP>

VNC. , IP- , :

: 5901

, vnc, , , , , vncservers

, , , Xfce xstartup.

4.

, sftp'ing- - . , , .

EFS, , . Amazon . , . , , , EFS. EFS ( NFS_Mount), TCP- 2049, , MasterVNC. , , MasterVNC . EFS .

EFS:

sudo mkdir /mnt/YOUR_MOUNT_POINT_DIR
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-YOUR_EFS_ID.efs.us-east-1.amazonaws.com:/ /mnt/YOUR_MOUNT_POINT_DIR

/mnt/YOUR_MOUNT_POINT_DIR 6 , 1 2, . , passwd , vncpasswd. ~/.vnc/passwd.

/mnt/YOUR_MOUNT_POINT_DIR/bin/installGui.sh/mnt/YOUR_MOUNT_POINT_DIR/bin/installXfce.sh

/mnt/YOUR_MOUNT_POINT_DIR/rsrc/vnc/vncservers-ec2-user/mnt/YOUR_MOUNT_POINT_DIR/rsrc/vnc/xstartup/mnt/YOUR_MOUNT_POINT_DIR/rsrw/vnc

/mnt/YOUR_MOUNT_POINT_DIR/rsrc/yum/yum.repos.d/centos.repo

. , ( MasterVNC), ssh , EFS installGui.sh.

5.

1 , CLI AWS . EFS installGui.sh, AWS CLI. CLI.

, . EC2 EMR . , , EMR , ( ).

EC2 --user-data. , .

aws ec2 run-instances --user-data file:///PATH_TO_YOUR_SCRIPT/top.sh  ... other options

EMR . S3 URI .

aws emr create-cluster --bootstrap-actions '[{"Path":"s3://YOUR_BUCKET/YOUR_DIR/top.sh","Name":"Custom action"}]' ... other options

, top.sh , - , , EC2 EMR. , 3 . , "run-if" . run-if , S3.

. , , - : /var/log/cloud-init-output.log. , .

top.sh

#!/bin/bash

# note: conditional bootstrap function run-if has a bug, workaround ...
# this function adapted from https://forums.aws.amazon.com/thread.jspa?threadID=222418
# Determine if we are running on the master node.
# 0 - running on master, or non EMR node
# 1 - running on a task or core node

check_if_master_or_non_emr() {
    python - <<'__SCRIPT__'
import sys
import json

instance_file = "/mnt/var/lib/info/instance.json"

try:
    with open(instance_file) as f:
        props = json.load(f)
    is_master_or_non_emr = props.get('isMaster', False)

except IOError as ex:
    is_master_or_non_emr = True   # file will not exist when testing on a non-emr machine

if is_master_or_non_emr:
    sys.exit(1)
else:
    sys.exit(0)
__SCRIPT__
}

check_if_master_or_non_emr
IS_MASTER_OR_NON_EMR=$?

# If this machine is part of EMR cluster, then ONLY install on the MASTER node

if [ $IS_MASTER_OR_NON_EMR -eq 1 ]
then
    sudo mkdir /mnt/YOUR_MOUNT_POINT_DIR

    sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-YOUR_EFS_ID.efs.us-east-1.amazonaws.com:/ /mnt/YOUR_MOUNT_POINT_DIR

    . /mnt/YOUR_MOUNT_POINT_DIR/bin/installGui.sh
fi

exit 0