7 Software release tools you should know

DevOps

You can do your release management as simple as command-line: mvn release:prepare;mvn release:perform.

However, there are lots of tools that make your release great and pleasant.
1. jira
Your project manager and scrum master can manage the release process with jira. A jira can integrate with bitbucket, slack, confluence etc. You can branch out github code base or create pull request in jira. Whenever you checked in code or created pull request, the jira will show updates; your co-worker can even get email when their name get mentioned or pull request is assigned to them. You can created all sorts of jiras: story, task, deployment, bug, feature, incident, etc. These jiras can have all sorts of relationships: subtask, related to, block, epics, etc. Isn't that pleasant?

2. bitbucket, github
You can do all git action with command line, however git management tools such as bitbucket makes your git usage much more efficient. In bitbucket, you can do basic operations such as create repository, create branches, create pull request, code review, merge, etc. It can do more than that. When bitbucket is integrated with bamboo or jenkins, your code check-in will trigger auto-build, and the build result can be a factor in the merge criteria. Besides, your merge will trigger auto version update, isn't it magical? Whenever you want, you can get lots of information by searching code base, lookup pull requests, filter on pull requests, compare tagged versions, and lots more.

3. sonatype nexus
Maven central repository allow your maven project to find dependencies. However, if you want to manage your own private jar collections, your may need a repository manager server like sonatype nexus. It allows you to proxy, collect, and manage your dependencies. With a few lines about the repo server url and credential added into ~/.m2/settings.xml file, the developers can publish to your sonatype nexus server hosted repos, this will also allow your mvn to use your repo as a mirror of central.

4. IC tools
Tools such teamcity, jenkins, electric cloud, bamboo act as the continuous integration build server. These build servers tie automated builds, tests, and releases together in a single workflow. For example, when bamboo is integrated with bitbucket, code check in will trigger bamboo builds. Some builds create snapshot artifacts, others create released artifacts. With the gui, you can create new build project, set up the trigger criteria, specify the git repo, specify the test/release command such as
mvn release:prepare;mvn release:perform;
You can even setup the quality gates by adding Nexus IQ and SonarQube etc. as release gate tasks.

5. Nexus IQ
You don't want to have code vulnerability to be in production, so you can setup Nexus IQ tool to scan and find bad codes. The Nexus IQ setup gui allows you to customize policy, so you can define which components are acceptable, and which are not.

6. SonarQube
Besides code coverage, SonarQube can do more such as finding smelly code or bad style, so your code is not only reliable but also smells good.

7. Automatic deployment tools
After IC tools finished test and release, they will create artifacts that need to be deployed on various servers on your DEV, QA, PILOT or PROD environment. You can use scp commands to copy the artifact such as jar, war, zip, tar.gz files on the target server's web-service (such as tomcat/jboss) directory. However, you don't have to do this kind of routine manually. Your IC tool can trigger an automatic deployment job as one of the release task. For example, automatic deployment tools such as RunDeck, Puppet, bladelogic etc. can execute user defined deployment jobs on server nodes. These nodes are your unix servers located on different environment. The artifacts are push to those nodes automatically at the end of the IC process.

maven trouble-shooting checklist

In theory, in order to use maven to build a project, all you need to do is to download maven, then update ~/.bashrc to have maven bin in $PATH.

export PATH=/Users/youruser/Downloads/apache-maven-3.2.5/bin/:$PATH

However, in reality, sometimes your maven gets stuck when downloading dependencies. Don't blame yourself for fat fingers, for big projects, the maven build process different every single time! First of all, your dependency could have snapshot versions, which might get updated on the repo 5 minutes ago, so when maven downloaded the latest one, trouble is introduced. Even though your direct dependency have no snapshot version. Your indirect dependency could have snapshot version dependency, downloading them also brings new jars which brings trouble into your build process. Besides, your OS/network admin could applied rules on your account, so that some of your remote repos can no longer be reached. Your build will still go on because your cached jar in local repo, however the cached version is now differ from the latest version, so your build process will differ from your coworkers' build process. Even though nothing happens in the environment, your download could be erroneous during file transfer due to network package loss, memory buffer, stale file stub etc.

Here are a trouble-shooting checklist.

1. find which maven is used.

which mvn

This command also tells you the maven home location.

There are two locations where a settings.xml file may live:
The Maven install: ${maven.home}/conf/settings.xml
A user’s install: ~/.m2/settings.xml
The user's install takes precedence than the other.

check both settings.xml file to make sure what properties are override. Without override, you default maven repo should locate inside ~/.m2/repository

2.  find if the complained jar file is in your local and remote repo or not.
The maven build will check local cache first, then go to download the remote repo for snapshot versions and won't download the release versions.
If the jar file is in the local repository ~/.m2/repository. It could be corrupted during download. You can use
mvn clean -U package
to force maven to re-download the existing local repository files from remote repository.
To achieve the same goal, you can also delete that dependency folder, or even the entire ~/.m2/repository folder, so that maven can not find the jar in local cache and have to trip to the remote repository to re-download everything.

Notice often times, the maven is complain about a jar and its depended jar file chains -- check each one of them, not just the first one in the complained list!

If you are helping others to trouble shoot build issue and the complained jar files are in your local repo but not in the remote repo. Chances are it just builds for you but won't build for others. Because someone has deleted the needed jar files in the remote repository, leaving a bug in the unaware victim code base.

3. restart you computer if possible. It will fix weird issues related to memory and network policy etc. For example network policy changes when you work from home with vpn then work in office.

4. check eclipse maven settings.

in eclipse -> preferences -> maven -> user settings, make sure you didn't override any default settings, so that you are using the default local repository. If you don't want to rely on default, literally specify the default location so that you are sure.

right click the project, then "maven", then "update project..", you can check the force update check box to have eclipse do the equivalent of
mvn clean -U package

5. restart eclipse, eclipse could run into weird issue internally.

6. ask if other people has the same build issue, then ask them to send you the ~/.bashrc, ~/.bash_profile, ~/.m2/settings.xml, and the /path-to-maven-home/conf/settings.xml. Also ask for the result of ls -lrt ~/.m2/repository/path/to/complained/jar

7. if you are using git or svn, find out who added the dependency that causes the trouble, have a conversation about it.

svn to git migration

Migrating svn to git needs some plan. There is a period, when the new git repo is created but the developers are continue to commit to the old svn repo.

Bitbucket provided a tool called svn-migration-scripts.jar to make the process easy. Down load the jar file and put under home directory ~/.

5 steps migration.

step 1. create a case-insensitive file-system image. 

If you are using linux such as redhat, ubuntu, your system already case-sensitive. If you are using mac os X, your system is using case-insensitive file-system. Run the following command to tell:

java -jar ~/svn-migration-scripts.jar verify

If the output looks like this, you then need to create a image.
"You appear to be running on a case-insensitive file-system. This is unsupported, and can result in data loss."

java -jar ~/svn-migration-scripts.jar create-disk-image 5 GitMigration

once the directory is created, cd there

cd ~/GitMigration

step 2. Extract author information

java -jar ~/svn-migration-scripts.jar authors https://svn.example.com > authors.txt

modify the emails in authors.txt

step 3. clone the svn to git repo

if your svn don't have a trunk, branch, tag lay out, use the following command.

git svn clone -authors-file=authors.txt https://svn.yoursvnrepo.com/YourSvnProj YourSvnProjAsGit

otherwise, you can specify the location of your trunk, branches, tag, use the following command instead

git svn clone --trunk=/trunk --branches=/branches --branches=/bugfixes --tags=/tags --authors-file=authors.txt https://svn.yoursvnrepo.com/YourSvnProj YourSvnProjAsGit

now clean the created project.

java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git --force

step 4. upload to git repo

Once you created a new git repo in bitbucket, you can push your local repo to the remote repo.

git remote add origin https://<user>@bitbucket.org/<user>/<repo>.git

git push -u origin --all

git push --tags

step 5. start to use git repo

now your coworker can start to clone the git repo and work on it.

git clone https://<user>@bitbucket.org/<user>/<project>.git <destination>

for the users that are still committing to the old svn, use the following command to synchronize

git config svn.authorsfile <path-to-authors-file>
git svn fetch
java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar sync-rebase
java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git --force

This Ocean Waves Power Plant is The First of Its Kind In All of Latin Am...

AI found Mysterious fast radio bursts from deep space, ‘could be aliens’

Canadian-led team of astronomers searching for FRBs have recently detected a repeating FRBs that could be alien signal. With the Breakthrough Listen program used AI (artificial intelligence), they have found more than 60 FRBs, two of the FRBs was detected repeating.

The origin of fast radio bursts (FRBs), millisecond-long pulses of radio waves, may be generated by black holes or super-dense neutron stars merging together. However, for such astrophysical phenomena events randomly repeated at the same location twice has a tiny chance.

search keywords: FRBs, alien

First Commercial Quantum Computing System from IBM

IBM debuts the Q System One, an integrated quantum computing system made for commercial and research use, the first of its kind in the world.

The system can be used to predict events that are almost impossible for classical computing, but which quantum computing is uniquely adept at solving. For example, predict financial events, optimize logistic problems.

𝑖ℏ∂Ψ(𝑞,𝑡)∂𝑡=HΨ(𝑞,𝑡)
Quantum mechanics describe our world as entangled waves in time and space (or a set of vectors where time and space dimensions are just mathematical presentations), instead of collection of small particles moving in space. This means that, because of this property of measurement in quantum mechanics, our universe is inherently a non-local, and everything seems connected. In another word, any quantum mechanics interaction has the whole universe considered, that's why it is simpler to predict financial events for quantum mechanics -- complexity and considering the influence of outside world is not a problem at all in quantum mechanics!

Here is an example program I ran on a 3 qubits IBM Q quantum computer. It demonstrate the quantum entanglement.

entanglement demonstration on IBM Q quantum computer

Entanglement is the weirdest of all quantum phenomena. Two or more quantum objects are entangled when, despite being too far apart to influence one another, they behave in ways that are 1) individually random, but also 2) too strongly correlated to be explained by supposing that each object is independent from the other. As the basic phenomena of physics, entanglement is everywhere. Some particles or waves in your body and my body might be in entangled states now, they go hand in hand. It is mind blow -- when your brain managed to flip one wave in your body, it flipped the other half in mine too, even though I never know you. Yes, no matter who we are, our thought litterally changed the world.

See the demo result? We only got 11 and 00 states, there is no 10 and 01 states. The 2 particles always agree with each other. In theory, the two qubits always agree, even you move one of them zillions of miles away.

include "qelib1.inc";
qreg q[2];
creg c[2];

h q[0];
cx q[0],q[1];
measure q[0] -> c[0];
measure q[1] -> c[1];

entanglement demonstration

search keyword: IBM CES 2019 First Integrated Quantum Computing System, IBM Q

slack short cuts and commands

Slack is a useful communication tool. You can have direct chat with a contact, host a group chat, open a channel for topic discussion. It can also be integrated with other tools/apps such as zoom, splunk, jira, ssh, bitbucket, etc, allowing you to host video conference, share screen, share files, run reports etc.

There are magic in slack realm, here are the spell syntax:

Command + / list all the keyboad shortcuts for Slack.

For example:

• command + [    previous channel or direct message window visited.
• command + ]    next channel or direct message window visited
• shift + enter will create a new line in your message
• command + U will upload a file
• command + shift + enter will add a new code snippets
• command + +    will zoom in
• command + -     will zoom out

/ will list the slack apps you can invoke. 

For example:

• /zoom                 will send a zoom meeting to your contacts
• /giphy words      will send a cool image relevant to the word to the chat window
• /poll "Poll question?" "Option1" "Option2"       will create a poll in your chat window

command + shift + G

command + shift + G is a handy shortcut.
In eclipse IDE, it means search the reference of a method in workspace, which is similar to Control + Option + H.

In mac os x finder, it means go to directory.

A useful usage scenario is as follows:

You look up a java method's references in workspace by typing command + shift + G, then  find an interesting class. You want to share the file to your coworker. So you right click the file, select properties, find the path in location, copy the path.

You then click finder icon or type command + space to open spotlight then type finder to open the finder.

Next you type command + shift + G to invoke "go to directory" UI, then paste the path and type enter. The finder will land on the file you want to send.

You then hightlight the file then drag it into the slack window to share it.

7 useful mac os x command line tools

Mac OS X has Darwin Unix like os. There are command line tools you can use in the console.

1. brew is a command to install other command line tools for mac os x.
To install brew and add the brew binary into the PATH, run the following commands in the console.

mkdir homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C homebrew;

echo export PATH=`pwd`/homebrew/bin:'$PATH' >> ~/.bashrc;

source ~/.bashrc;

which brew;

The last command "which brew;" should give the full path to brew binary file.

Now you have brew command available, you can install other command line tools easily.

2. pigz is a tool that is gzip equivalent but runs faster by exploiting parallel computing. pigz is useful when you need to unzip contents with gzip or zlib format. For example, you rabbitmq messages might be zlib compressed, then base64 encoded. Your decoding commands might looks funny:
base64 -D input.txt | pigz -d -z -c

To install pigz, issue the following command:
brew install pigz

3. apache bench (ab) is a command line tool that allows you to bench mark web sites. The most useful flag is -c, which allows you to specify the number of concurrent http requests sent to the host each time.
For example:

ab -n 100000 -c 10 -t 30 http://xyzcode.blogspot.com/

the above commands send total 10000 requests to xyzcode.blogspot.com, each time, 10 concurrent requests are sent.

ab is installed by default on mac os x, type which ab to check out. In case it is missing, you can install it with command:
brew install ab

4. pybot is a command line tool that automate stuff like ssh, web browsing etc. It is equivalent to have a robot to type the keyboard and read the screen then act upon it for you.

To install pybot, issue the following command:
brew install robot-framework

Since brew also installed python's package management tool, you can also use pip to install pybot:
pip install robotframework

5. in case pybot is an overkill for you, you can use expect command to automate ssh chat with remote hosts like a robot. Expect commands expect input, then send the response without any user interaction.
expect script is installed by default, check the installation path with
which expect;

6. zgrep is a better tool than grep.
zgrep allows you to grep content in zip files and normal files. It will be useful to search something in current log and backup logs.
The flexible syntax really compliments your searching strategy. So if you know the log will looks like

2018-11-07 16:42:25-05 HostNameppp Sophos Installer[72881]: [SGCCDFSBroker.m:306] Feedback json file was successfully uploaded (status code: 201).

Your searching strategy could be:
zgrep -i 'Sophos.*json.*successful' /var/log/install.log

each .* separated string became a filter string, so that you can narrow down the search.

You can use or syntax like:
zgrep -i 'Sophos.*json.*successful' /var/log/install.log | grep --color '72881\|72889'

zgrep is installed by default
which zgrep; will give you the installation location.

7. mysql is a command line tool to query mysql database. It allows you scripting mysql queries. You can pipe the result into zgrep to search relevant information.

mysql --host mysql.local -uuser -ppass -e 'select * from ct_prov.Student\G'

to install mysql command line tool
brew install mysql

7 tips for text edit efficiency

1. Inside chrome, use option + command + left to move tab by tab from left to right, use option + command + right to move tab by tab from right to left.

2. Inside chrome, right click then click inspect, then click network. Refresh the page to view the page loading process.

3. install chrome tools to handle json, xml etc.
jsonEditer to tidy and modify json string
https://chrome.google.com/webstore/search/jsonEdit?hl=en

ooxml tool to edit and compare xml files
https://chrome.google.com/webstore/detail/ooxml-tools/bjmmjfdegplhkefakjkccocjanekbapn?hl=en

4. install diffmerger to compare files side by side
https://sourcegear.com/diffmerge/downloads.php

5. use --color option to highlight text (good for your eye), use --text option to avoid searching missing in zip files with special characters.
>zgrep KEY --color --text demoClient.key
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----

6. In xterm2, if you get a huge file which can not be displayed on one screen, click Session -> Edit Session -> Terminal -> then modify scroll back lines to a bigger number.

7. In xterm2, if you want to view several terminals side by side, type command + D to split horizontally, click command + shift + D to split vertically, then type command + shift + i to type in all the terminals. Type command + shift + i again to negate the effect.