[nayuos] Consolidate changes from Isabelle and Tristain

-  improve bashrc file
-  drop nodejs package which was causing troubles
-  add nayuos policy for duckduckgo to be the default search engine
-  refactor build script

software/nayuos/README.md

 # NayuOS
 
-This is a SlapOS recipe to build NayuOS. It needs to be put in the <code>/srv/slapgrid/\<part\>/srv/runner/project/slapos/software/</code> directory. The created directory is called <code>\<nayuos_build_dirname\></code> in this documentation.
+This is a SlapOS recipe to build NayuOS.
 
 ## License
 
@@ -8,42 +8,61 @@ GPL v2 or later
 
 ## Requirements
 
-*   sudo on the host (for now)
-*   some environment variables need to be authorized to be propagated when cros_sdk calls sudo:
+*   sudo on the host
+*   environment variables need to be authorized to be propagated when cros_sdk calls sudo: `: Defaults        env_keep += "CROS_CACHEDIR DEPOT_TOOLS"`
+*   slapuser with sudo rights to execute the cros_sdk scripts (needed to access the chroot environment provided by Chromium OS)
+in /etc/sudoers (replace slapuser9 by your user, and release-R48-7647.B by the release you have chosen): `: slapuser9    ALL=NOPASSWD: /srv/slapgrid/slappart9/srv/runner/instance/slappart0/parts/chromiumos/release-R48-7647.B/chromite/bin/cros_sdk, /srv/slapgrid/slappart9/srv/runner/instance/slappart0/wrapper_bin/wrapper_cros_sdk, /bin/kill`
 
-: Defaults        env_keep += "CROS_CACHEDIR DEPOT_TOOLS"
+It's useful to have the right to kill cros_sdk processes, when needed. ;)
 
-*   slapuser with sudo rights to execute the cros_sdk scripts (needed to access the chroot environment provided by Chromium OS)
-in /etc/sudoers (replace slapuser9 by your user, and release-R48-7647.B by the release you have chosen):
+## Technical notes
 
-: slapuser9    ALL=NOPASSWD: /srv/slapgrid/slappart9/srv/runner/instance/slappart0/parts/chromiumos/release-R48-7647.B/chromite/bin/cros_sdk, /srv/slapgrid/slappart9/srv/runner/instance/slappart0/wrapper_bin/wrapper_cros_sdk, /bin/kill
+After any change to the build process it is necessary to delete (using sudo)
+the building environment at `~/srv/runner/instance/slappart0/parts/chromiumos/<TAG>`.
+
+BEWARE that the web runner is serving images for the [official website](https://nayuos.nexedi.com).
+
+NayuOS and ChromiumOS is "just" a version of Gentoo. Thus it uses `ebuild` packages
+and anything installable in Gentoo can be installed to NayuOS too. Of course only
+during OS build phase and one has to count with limited space.
+
+### Upgrading (building new image)
+
+Please read **Requirements** section carefully. After selecting your desired 
+`release` from the list <https://chromium.googlesource.com/chromiumos/manifest/+refs>
+it is **necessary** to add `sudo` rules for that release as shown there.
 
-It's useful to have the right to kill cros_sdk processes, when needed. ;)
 
 ## Input
+
 In the vifib parameters (softinst\<nb\>.host.vifib.net \> Services \> Parameters):
 
-*   board / ex: peppy, swanky, ... (choosing daisy will accept all licenses for the daisy board build only, in order to use Mali drivers, see [chromium mailing list](https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-dev/Pf9ZG2itxWM))
-*   branch / ex: release-R46-7390.B (you can find the release in the [Chromium OS source tree](https://chromium.googlesource.com/chromiumos/manifest/+refs))
-*   keep_cache / yes|no (choosing "no" saves about 15Go of disk space per board, choosing "yes" will makes next build faster and less expensive in term of needed ressources because of not rebuilding everything)
+*   **board** / ex: peppy, swanky, ... (choosing daisy will accept all licenses for the daisy board build only, in order to use Mali drivers, see [chromium mailing list](https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-dev/Pf9ZG2itxWM))
+*   **branch** / ex: release-R46-7390.B (you can find the release in the [Chromium OS source tree](https://chromium.googlesource.com/chromiumos/manifest/+refs))
+*   **keep_cache** / yes|no (choosing "no" saves about 15Go of disk space per board, choosing "yes" will makes next build faster and less expensive in term of needed ressources because of not rebuilding everything)
 
 
 ## Output
-The image will be produced in:
-<code>/srv/slapgrid/\<part\>/srv/runner/instance/\<inst_part\>/parts/chromiumos/images/</code>
-and the logs are in:
-<code>/srv/slapgrid/\<part\>/srv/runner/instance/\<inst_part\>/var/log/cros_sources_dl.log</code> and <code>/srv/slapgrid/\<part\>/srv/runner/instance/\<inst_part\>/var/log/cros_build.log</code>
 
-The script that download the sources and build is located in
-<code>/srv/slapgrid/\<part\>/srv/runner/instance/\<inst_part\>/etc/run</code>
+Software release produces a build script `<instance_partition>/etc/run/cros_full_build`.
+
+Build produces
+
+*  Image: `<instance_partition>/parts/chromiumos/images/`
+*  Compilation logs: `<instance_partition>/var/log/cros_sources_dl.log`
+*  Build logs: `<instance_partition>/var/log/cros_build.log`
+
 
 ## External documents
 
-*   [ <code>repo</code> command reference ](https://source.android.com/source/using-repo.html)
-*   [ NayuOS official website ](https://www.nayuos.com)
+*   [ `repo` command reference ](https://source.android.com/source/using-repo.html)
+*   [ NayuOS official website ](https://nayuos.nexedi.com)
 *   [ crouton for chroot ](https://github.com/dnschneid/crouton) ([warning about verified boot](https://github.com/dnschneid/crouton/blob/2a1fc9da380650f47e2bcf37d00962bfb68c4830/installer/main.sh#L517-L536))
 
 ## Notes for possible improvements
+
 *   [ Running virtual machines on your chromebook ](https://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices/running-virtual-machines-on-your-chromebook)
 *   to have a more common User Agent (the one of ChromiumOS/NayuOS is quite rare and identifies the user, see [studies of the EFF](https://panopticlick.eff.org/static/browser-uniqueness.pdf)), it seems possible to change the User-Agent flag for guest mode in the getOffTheRecord function, and adding a line (key "kUserAgent" , value "some common user agent" string). Then rebuild Chromium and [add it to NayuOS](https://www.chromium.org/chromium-os/developer-guide#TOC-Making-changes-to-the-Chromium-web-).
 *   remove need of root priviledge for entering the chroot, maybe by using fakeroot in 'scripts/wrapper_sudo.in'?
+*   change more options on Chromium OS "Privacy" part by default: there are [a few options](https://support.google.com/chromebook/answer/114836) which still use Google services
+*   provide ChromiumOS package manager [ chromebrew ](https://skycocker.github.io/chromebrew/) by default

software/nayuos/instance.cfg

 [buildout]
 parts =
   parameters
+  promise-sudo-on-host
   template-full-build-script
   template-sudo-wrapper
   template-cros-sdk-wrapper
 
-# eggs given by software.cfg
+# standard declaration of eggs directories
 eggs-directory = {{ eggs_directory }}
 develop-eggs-directory = {{ develop_eggs_directory }}
 
@@ -16,6 +17,7 @@ develop-eggs-directory = {{ develop_eggs_directory }}
 recipe = slapos.recipe.build:gitclone
 repository = https://chromium.googlesource.com/chromium/tools/depot_tools.git
 branch = master
+git-executable = {{ git_path }}/git
 
 [customize-path]
 # add depot tools directory (for cros_sdk binary among others) and git directory to the path
@@ -25,6 +27,8 @@ command =
 [nayuos-ebuilds]
 recipe = slapos.recipe.build:gitclone
 repository = https://lab.nexedi.com/nexedi/nayuos-ebuilds.git
+branch = master
+git-executable = {{ git_path }}/git
 
 ############################################################################################
 
@@ -44,6 +48,7 @@ cert = ${slap_connection:cert_file}
 recipe = slapos.cookbook:mkdirectory
 log = ${buildout:directory}/var/log
 run = ${buildout:directory}/etc/run
+promise = ${buildout:directory}/etc/promise
 wrapper_dir = ${buildout:directory}/wrapper_bin
 cros_location = ${buildout:directory}/parts/chromiumos
 ebuilds_dir = ${nayuos-ebuilds:location}
@@ -55,6 +60,13 @@ wrapper_cros_sdk=${directory:wrapper_dir}/wrapper_cros_sdk
 bash=/bin/bash
 sudo=/usr/bin/sudo
 
+[promise-sudo-on-host]
+# assert sudo is installed on the slapos host, as it is required to enter the
+# chroot 'cros_sdk'
+recipe  = slapos.cookbook:wrapper
+wrapper-path = !py! '${directory:promise}/' + '${:_buildout_section_name_}'[8:]
+command-line = sudo -V
+
 [template-sudo-wrapper]
 recipe = slapos.recipe.template:jinja2
 template = {{ scripts_dir }}/wrapper_sudo.in
@@ -75,7 +87,7 @@ mode = 0700
 context =
 # XXX bash path is the one from the host
   key bash_path bin:bash
-  raw git_path {{ git_path }}
+  raw git_path  {{ git_path }}
   raw curl_path {{ curl_path }}
 
 [template-full-build-script]
@@ -84,17 +96,18 @@ context =
 recipe = slapos.recipe.template:jinja2
 template = {{ scripts_dir }}/cros_full_build.in
 rendered = ${directory:run}/cros_full_build
-md5sum = 75599e6b8418a5f3756c7c7b26600399
+md5sum = 2bb9dd83260ea96dd6a6602f6faa9794
 mode = 0700
 context =
-  key bash_path bin:bash
-  key instance_log_dir directory:log
-  key cros_location directory:cros_location
-  key export_path_cmd customize-path:command
-  key branch parameters:configuration.branch
-  key boards_list parameters:configuration.boards
-  key keep_cache parameters:configuration.keep_cache
-  key ebuilds_dir directory:ebuilds_dir
-  key scripts_dir directory:scripts_dir
-  key logo_dir directory:logo_dir
-  raw nayu_dev_packages net-libs/nodejs net-misc/re6stnet dev-vcs/git dev-python/flask dev-python/virtualenv sys-fs/cryptsetup
+  key bash_path                         bin:bash
+  key instance_log_dir                  directory:log
+  key cros_location                     directory:cros_location
+  key export_path_cmd                   customize-path:command
+  key branch                            parameters:configuration.branch
+  key boards_list                       parameters:configuration.boards
+  key keep_cache                        parameters:configuration.keep_cache
+  key ebuilds_dir                       directory:ebuilds_dir
+  key scripts_dir                       directory:scripts_dir
+  key logo_dir                          directory:logo_dir
+  raw nayu_dev_rootfs_packages          app-misc/nayuos-chromium-policy
+  raw nayu_dev_packages                 net-misc/re6stnet dev-vcs/git dev-python/flask dev-python/virtualenv sys-fs/cryptsetup

software/nayuos/scripts/test_nayuos_image

@@ -14,8 +14,10 @@ ORIGINAL_GRANDENET_SCRIPT=~/trunk/src/third_party/chromiumos-overlay/net-misc/re
 GRANDENET_SCRIPT="usr/local/bin/grandenet"
 INIT_SSH_SERVER="etc/init/openssh-server.conf"
 BASHRC="etc/skel/.bashrc"
-EXPECTED_ALIAS="alias git='git --exec-path=/usr/local/libexec/git-core/'"
+GIT_CORE="usr/local/libexec/git-core/"
+GIT_EXPECTED_EXPORT="export GIT_EXEC_PATH=/usr/local/libexec/git-core"
 VIRTUALENV_BIN="usr/local/bin/virtualenv"
+CHROMIUM_POLICY="etc/chromium/policies/recommended/nayuos_policy.json"
 
 HAS_FAILED=0
 
@@ -32,14 +34,17 @@ function print_result() {
 
 # MOUNT IMAGE AND GET INFO
 
-install -d ${MOUNTPOINT}
+install -d "${MOUNTPOINT}"
 ./mount_gpt_image.sh --safe -f $( ./get_latest_image.sh --board=${BOARD} ) -r ${MOUNTPOINT}
 echo $(ls "${MOUNTPOINT}/usr/local")
 if [[ $(ls "${MOUNTPOINT}/usr/local") ]] ; then
-  my_diff=$(diff ${ORIGINAL_GRANDENET_SCRIPT} "${MOUNTPOINT}/${GRANDENET_SCRIPT}")
+  my_diff=$(diff "${ORIGINAL_GRANDENET_SCRIPT}" "${MOUNTPOINT}/${GRANDENET_SCRIPT}")
   opensshd_config=$(ls "${MOUNTPOINT}/${INIT_SSH_SERVER}")
-  gitalias=$(grep "${EXPECTED_ALIAS}" "${MOUNTPOINT}/${BASHRC}")
+  gitcore=$(ls "${MOUNTPOINT}/${GIT_CORE}")
+  gitexport=$(grep "${GIT_EXPECTED_EXPORT}" "${MOUNTPOINT}/${BASHRC}")
   virtualenvbin=$(ls "${MOUNTPOINT}/${VIRTUALENV_BIN}")
+  chromium_policy=$(ls "${MOUNTPOINT}/${CHROMIUM_POLICY}")
+
 else
   is_empty=1
 fi
@@ -50,7 +55,7 @@ rmdir ${MOUNTPOINT}
 # PRINT RESULTS
 
 echo "* test if /usr/local exists"
-if [[ ${is_empty} == 1 ]] ; then
+if [[ "${is_empty}" == "1" ]] ; then
   print_result ${FAILURE} "/usr/local is empty."
 else
   print_result ${SUCCESS} "/usr/local is not empty."
@@ -71,10 +76,18 @@ else
     print_result ${SUCCESS} "opensshd config removed."
   fi
 
-  echo "* test git quick fix for option --exec-path"
+  echo "* test if git core directory is not empty"
+
+  if [[ "${gitcore}" == "" ]] ; then
+    print_result ${FAILURE} "No git file in ${MOUNTPOINT}/${GIT_CORE}."
+  else
+    print_result ${SUCCESS} "git core directory exists and is not empty."
+  fi
+
+  echo "* test bashrc changes for git paths quick fix"
 
-  if [[ ${gitalias} == ""  ]] ; then
-    print_result ${FAILURE} "Expected alias for git command not in ${MOUNTPOINT}/${BASHRC}. Should be: ${EXPECTED_ALIAS}"
+  if [[ "${gitexport}" == ""  ]] ; then
+    print_result ${FAILURE} "Expected alias for git command not in ${MOUNTPOINT}/${BASHRC}. There should be: ${GIT_EXPECTED_EXPORT}"
   else
     print_result ${SUCCESS} "git alias is correct."
   fi
@@ -86,6 +99,14 @@ else
   else
     print_result ${FAILURE} "Expected virtualenv binary not in ${MOUNTPOINT}/${VIRTUALENV_BIN}."
   fi
+
+  echo "* test if Chromium policies are installed"
+
+  if [[ ${chromium_policy} ]] ; then
+    print_result ${SUCCESS} "Chromium policies are installed."
+  else
+    print_result ${FAILURE} "Expected Chromium policies file not in ${MOUNTPOINT}/${CHROMIUM_POLICY}."
+  fi
 fi
 
 exit ${HAS_FAILED}

software/nayuos/software.cfg

@@ -6,19 +6,19 @@ extends =
 parts +=
 # use stack/slapos.cfg
   slapos-cookbook
-  verify-packages-exist-on-host
   template-instance
-  install-eggs-for-the-instance
+
+# 2017-06-28: XXX is this really necessary? Remove after successful build
+#add eggs needed by the instance
+#find-links += 
+#  https://pypi.python.org/packages/source/s/slapos.recipe.build/slapos.recipe.build-0.21.tar.gz
+#  https://pypi.python.org/packages/source/p/plone.recipe.command/plone.recipe.command-1.1.zip
+
+versions = versions
 
 [versions]
 slapos.recipe.template = 3.0
 
-[verify-packages-exist-on-host]
-recipe = plone.recipe.command
-stop-on-error = true
-# run the same command when installing and on updates
-update-command = ${:command}
-command = sudo -V
 
 [template-instance]
 # create the instance.cfg file in the buildout directory
@@ -26,24 +26,17 @@ command = sudo -V
 recipe = slapos.recipe.template:jinja2
 template = ${:_profile_base_location_}/instance.cfg
 rendered = ${buildout:directory}/instance.cfg
-md5sum = be6bd665c0b356a84090d200ea14b33c
 mode = 0644
+md5sum = 085560ea84a5a7f633de0ef63a6ba7aa
 scripts_dir = ${:_profile_base_location_}/scripts
 logo_dir = ${:_profile_base_location_}/logo
 curl_path = ${curl:location}/bin
 git_path = ${git:location}/bin
 context =
-# for access to the eggs from the instance
-  key eggs_directory buildout:eggs-directory
-  key develop_eggs_directory buildout:develop-eggs-directory
-  key scripts_dir :scripts_dir
-  key logo_dir :logo_dir
-  key curl_path :curl_path
-  key git_path :git_path
+  key scripts_dir             :scripts_dir
+  key logo_dir                :logo_dir
+  key curl_path               :curl_path
+  key git_path                :git_path
+  key eggs_directory          buildout:eggs-directory
+  key develop_eggs_directory  buildout:develop-eggs-directory
 
-[install-eggs-for-the-instance]
-# after installation of the eggs, the recipes
-# will be available (added to sys.path)
-recipe = zc.recipe.egg
-eggs = slapos.recipe.build
-       plone.recipe.command