Setting up developer environment

From MMS Wiki
Jump to: navigation, search

Main setup

All sorts of warnings are normal in the steps below. Especially during npm and pip package installation and when running db:sync

Ubuntu 20.04

  • Install some apt packages
sudo apt update
sudo apt install -y git-core python3-dev curl libfontconfig1-dev libc6 libgc-dev ocaml make xdg-utils ffmpeg python3-pip fluidsynth graphicsmagick sox libsox-fmt-mp3 s3cmd nodejs npm libjpeg-dev

TODO: copy any missing required packages from first step of the next section.

  • Install core NPM packages globally
sudo npm install -g bower grunt-cli cordova
  • Install HaXe
https://haxe.org/download/linux/
  • Haxelib
haxelib setup
haxelib install lime
haxelib run lime setup
# ln -s hxMath.h hxMath.h.h # In the right directory
  • Clone git repos:
git clone git@bitbucket.org:matchmysound/MMS.git
cd MMS
# clone mms-haxe into MMS/haxe/
git clone git@bitbucket.org:matchmysound/mms-haxe.git haxe
  • TODO: Add virtualenv instructions
  • Install pip packages
pip3 install -r matchmysound/requirements.txt
  • Install bower packages
cd matchmysound
bower install
  • Install node (build) packages
cd ../grunt
npm install
cd ..

macOS

Incomplete instructions!

  • Install XCode

- Not command line tools. If you installed those, uninstall them and xcode-select -s the Xcode app

brew install graphicsmagick postgresql libsass openssl redis libav sox fluid-synth python
  • Install pip
easy_install pip celery
  • Install latest Java JDK to get java command
  • Do steps NPM, Haxelib, Git, pip, bower, node(build) from the Main setup for Ubuntu
  • Do steps Download Docker files, Set up DB from Local Development Server
  • Continue to "Internal Setup"

- pip should not be run with sudo, but with --user flag instead, if problems occur

haxelib 3,2,81 version needs to have a symlink like latest as cordova plugin xml parser has a bug which dies on commas in path name. Also, XCode project needs "Build settings" -> "Other linker flags" -> -stdlib=libc++ for compile to work.

You may also occasionally need to do

export IPHONE_VER=10.0

Windows 10 (WSL)

Possibly incomplete instructions!

First, follow this guide to set up WSL. Restart.

Then follow with the standard Linux installation (but make sure to put it somewhere in /mnt/)

Docker needs to be set up both inside WSL and in Windows proper. Standard docker install requires pro version for Hyper V support. To get around this, install "Docker toolbox" instead. Then do

docker-machine env

in PowerShell, and copy those lines over to the end of your .bashrc (changing path to /mnt/c/User/.... from C:\User\...)

While there, also add the line (required for pip binaries):

export PATH=$PATH:~/.local/bin

Now we also need to add the following to current_project.json:

"DOCKER_HOST":"192.168.99.100",
"HOSTNAME":"localhost.matchmysound.com"

If Bower gives you trouble,

chown <your_username> -r ~/.config

Node also needs

sudo apt-get install nodejs-legacy

If node-sass is acting up, do

npm rebuild node-sass

Hosts file for Windows is at C:\Windows\System32\Drivers\etc\hosts . To edit it, right-click notepad and choose "Run as Administrator" NB! Hosts file for Linux (/etc/hosts) needs the same update too!

Local development server

Ubuntu 20.04

  • Some packages
sudo apt install -y libpq-dev postgresql-client redis-tools
  • Docker setup
sudo apt install -y docker.io
sudo groupadd docker # May already exist
sudo usermod -aG docker $USER
  • Log out and in again
  • Download Docker files
docker pull redis
docker pull postgres
  • Some configuration
echo "127.0.0.1 localhost.matchmysound.com" | sudo tee --append /etc/hosts
  • Set up DB
./manage.py target matchmysound # Lack of output is correct
# ./manage.py start db # in another terminal or with &
./manage.py db:create
./manage.py db:restore backup.sql # Ask for a db backup file from Andres or Karl
  • Close the DB with CTRL+C

Optional

A useful thing to do is to make MMS/tmp/web folder into a ramfs disk. ~128m should suffice. /etc/fstab line for that is:

ramfs       ~/MMS/tmp/web ramfs   nodev,nosuid,noexec,nodiratime,size=128M   0 0

Usage

  • Start up the environment (2 separate terminal windows)
./manage.py start
./manage.py start client

Troubleshooting

  • If client is acting up, then try ./manage.py start db redis api celery instead of .manage.py start.
  • If you see ENOSPC error, it is a problem with too many inotify watches. Solution:
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
  • If you see sslwrap error, it is a gevent thing. 1.0.2 fixes it for Python 2.7.9 but if you have 2.7.8 that has the bug backported into it, just go to /usr/local/lib/python2.7/dist-packages/gevent/hub.py and change PYGTE279 to True.

Web app E2E tests

Setup

Install Protractor

sudo npm install -g protractor

Install Selenium browser drivers

sudo webdriver-manager update 

Usage

./manage.py test:run

Troubleshooting

If it complains about not finding them, you need to set the local npm to use the global protractor

cd grunt/node_modules; rm -rf protractor; ln -s /usr/local/lib/node_modules/protractor protractor

Mobile app E2E tests

Appium-based. iOS testing is only possible on macOS.

Ubuntu 18.04

Install Appium and its tool:

npm install -g appium appium-doctor

Check what steps remain to be done with that tool:

appium-doctor --android

Follow instructions from appium-doctor to setup prerequisites (Java; Android Studio; ~/.profile/~/.bashrc additions;).

Set up an AVD (Android Virtual Device). Default settings should be fine (Android 10). If you choose something different then you also need to change grunt/tests/wdio.conf.android.js.

macOS

  • Install Appium and its tool:
npm install -g appium appium-doctor
  • Check what steps remain to be done with that tool:
appium-doctor --ios

Some steps that might be mentioned in Appium Doctor:

  • Install Java
brew cask install java11
  • Install Android Studio:
brew cask install android-studio
  • Install carthage:
brew install carthage
  • Install some of the optional parts:
brew install ios-webkit-debug-proxy --HEAD
brew install libimobiledevice --HEAD
brew install usbmuxd --HEAD
brew install ios-deploy
brew install fbsimctl --HEAD
  • Install idb:
brew tap facebook/fb
brew install idb-companion --HEAD
pip3.7 install fb-idb

TODO: verify which of the preceeding are needed...

Usage

cd grunt
npm run test:android

MuseScore

Needed for creating exercises on your machine.

Clone repo:

git clone https://github.com/andrT/MuseScore.git ../MuseScore

Ubuntu

Then do "Install dependencies" and "Compiling the code" at:

https://musescore.org/en/developers-handbook/compilation/compile-instructions-ubuntu-14.10-git

macOS

(loosely based on https://musescore.org/en/developers-handbook/compilation/compile-instructions-macos-git)

Download Qt installer and use it to install Qt 5.4 (anything later won't work). Then:

brew install libogg libsndfile libvorbis pkg-config portaudio jack lame cmake git
PATH=$PATH:~/Qt/5.4/clang_64/bin
cd MuseScore
make -f Makefile.osx clean
make -f Makefile.osx
make -f Makefile.osx install

then copy mscore from MuseScore/applebuild directory to "Applications" (as you normally install stuff on OSX) And then create a symlink so it could be used from commandline with:

ln -s /Applications/mscore.app/Contents/MacOS/mscore /usr/local/bin/mscore

Android Dev

HaXe needs to be at least 3.3.0.

Ubuntu

  • You need to install gradle:
sudo apt install gradle
  • You also need Android Studio (which in turn requires JDK 1.8) from your software store or either of
sudo snap install android-studio --classic

or

flatpak install flathub com.google.AndroidStudio
  • Once that is installed, navigate to "Configuration" -> "SDK manager" and install:

"Android 8.0 (Oreo)" which has API level 26 in order to agree to some terms for Cordova.

  • Also, add these to your ~/.bashrc
export ANDROID_HOME=$HOME/Android/Sdk
export PATH=$PATH:$ANDROID_HOME/platform-tools
export PATH=$PATH:$ANDROID_HOME/tools
export PATH=$PATH:$ANDROID_HOME
export PATH=$PATH:$JAVA_HOME/bin

TODO: Verify which of the above ENV vars are required

Mobile app building

Fastlane-based. A more convenient way to build mobile apps.

Ubuntu

  • Add to ~/.bashrc:
export PATH=$PATH:$(ruby -e 'print Gem.user_dir')/bin
export MMS_DIR=$HOME/path-to-your-MMS-project
export LANG=en_US.UTF-8
  • Optionally also add there a wrapper to run fastlane more conveniently.
function fastlane() { cd $MMS_DIR/fastlane && bundle exec fastlane "$@"; cd ..; }

Without it all fastlane commands have to be prefixed with bundle exec and need to be run from $MMS_DIR/fastlane.

  • Install prerequisites:
sudo apt install ruby ruby-dev libcurl4-openssl-dev ruby-bundler
  • Install fastlane and its dependencies:
cd $MMS_DIR/fastlane && bundle update; cd ..
  • Set up all credentials (from Karl) in $MMS_DIR/mms/fastlane/credentials_do-not-commit/

Usage

  • To get a full list of available commands and WIP descriptions, run:
fastlane
  • Build and run on a connected Android device or a running emulator:
fastlane android build_and_run
  • Optionally give additional parameters via ENV vars instead of answering questions:
BUILD_TYPE=debug VERSION_NUMBER=6.6.6 MCONF=achievemusic fastlane android build_and_run

Editor (sublime)

Under Project - edit project, set

   "settings": {
       "expand_tabs_on_save": true,
       "trim_trailing_white_space_on_save": true
   }

Shortcuts:

  • Index project using search in ctrl-shift-p
  • goto definition using ctrl+win+alt+l
  • jump to definition win+ctrl+l

Set user configuration to ignore temporary dirs - Preferences - Package Settings - AngularJS - Settings - User

{
   "exclude_dirs":[
       "client/tmp",
       "client/.tmp",
       "client/node_modules",
   ]
}

Install Anaconda for python linting, goodies

Set user configuration in Anaconda to set max length a bit higher (note that you should still strive for 80)

{
   "pep8_max_line_length": 100
}

Set Anaconda User Key bindings

[
   {"command": "anaconda_goto", "keys": ["super+ctrl+alt+l"]},
   {"command": "anaconda_find_usages", "keys": ["super+alt+d"]},
   {"command": "anaconda_doc", "keys": ["alt+d"]},
   {"command": "anaconda_auto_format", "keys": ["super+a"]},
   {"command": "anaconda_goto", "keys": ["g", "d"],
       "context":
       [
           { "key": "selector", "operator": "equal", "operand": "source.python" },
           { "key": "setting.command_mode", "operand": true },
           { "key": "setting.is_widget", "operand": false }
       ]
   }
]

Shortcuts

  • goto definition using ctrl+win+alt+l
  • show docstring alt+d
  • find usages of function win+alt+d
  • autoformat file win+r (this cleans up a lot of the formatting-related pep8 warnings, good when refactoring and cleaning old code)