Tuesday, November 6, 2018

7 useful mac os x command line tools

Mac OS X has Darwin Unix like os. There are command line tools you can use in the console.

1. brew is a command to install other command line tools for mac os x.
To install brew and add the brew binary into the PATH, run the following commands in the console.

mkdir homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C homebrew;

echo export PATH=`pwd`/homebrew/bin:'$PATH' >> ~/.bashrc;
source ~/.bashrc;
which brew;

The last command "which brew;" should give the full path to brew binary file.

Now you have brew command available, you can install other command line tools easily.

2. pigz is a tool that is gzip equivalent but runs faster by exploiting parallel computing. pigz is useful when you need to unzip contents with gzip or zlib format. For example, you rabbitmq messages might be zlib compressed, then base64 encoded. Your decoding commands might looks funny:
base64 -D input.txt | pigz -d -z -c

To install pigz, issue the following command:
brew install pigz

3. apache bench (ab) is a command line tool that allows you to bench mark web sites. The most useful flag is -c, which allows you to specify the number of concurrent http requests sent to the host each time.
For example:

ab -n 100000 -c 10 -t 30 http://xyzcode.blogspot.com/

the above commands send total 10000 requests to xyzcode.blogspot.com, each time, 10 concurrent requests are sent.

ab is installed by default on mac os x, type which ab to check out. In case it is missing, you can install it with command:
brew install ab

4. pybot is a command line tool that automate stuff like ssh, web browsing etc. It is equivalent to have a robot to type the keyboard and read the screen then act upon it for you.

To install pybot, issue the following command:
brew install robot-framework

Since brew also installed python's package management tool, you can also use pip to install pybot:
pip install robotframework

5. in case pybot is an overkill for you, you can use expect command to automate ssh chat with remote hosts like a robot. Expect commands expect input, then send the response without any user interaction.
expect script is installed by default, check the installation path with
which expect;

6. zgrep is a better tool than grep.
zgrep allows you to grep content in zip files and normal files. It will be useful to search something in current log and backup logs.
The flexible syntax really compliments your searching strategy. So if you know the log will looks like

2018-11-07 16:42:25-05 HostNameppp Sophos Installer[72881]: [SGCCDFSBroker.m:306] Feedback json file was successfully uploaded (status code: 201).

Your searching strategy could be:
zgrep -i 'Sophos.*json.*successful' /var/log/install.log

each .* separated string became a filter string, so that you can narrow down the search.

You can use or syntax like:
zgrep -i 'Sophos.*json.*successful' /var/log/install.log | grep --color '72881\|72889'

zgrep is installed by default
which zgrep; will give you the installation location.

7. mysql is a command line tool to query mysql database. It allows you scripting mysql queries. You can pipe the result into zgrep to search relevant information.

mysql --host mysql.local -uuser -ppass -e 'select * from ct_prov.Student\G'

to install mysql command line tool
brew install mysql

Continue Reading »

Wednesday, October 31, 2018

7 tips for text edit efficiency

1. Inside chrome, use option + command + left to move tab by tab from left to right, use option + command + right to move tab by tab from right to left.

2. Inside chrome, right click then click inspect, then click network. Refresh the page to view the page loading process.

3. install chrome tools to handle json, xml etc.
jsonEditer to tidy and modify json string

ooxml tool to edit and compare xml files

4. install diffmerger to compare files side by side

5. use --color option to highlight text (good for your eye), use --text option to avoid searching missing in zip files with special characters.
>zgrep KEY --color --text demoClient.key

6. In xterm2, if you get a huge file which can not be displayed on one screen, click Session -> Edit Session -> Terminal -> then modify scroll back lines to a bigger number.
7. In xterm2, if you want to view several terminals side by side, type command + D to split horizontally, click command + shift + D to split vertically, then type command + shift + i to type in all the terminals. Type command + shift + i again to negate the effect.

Continue Reading »

Wednesday, October 17, 2018

How to exit java ExecutorService without calling shutdown method

The java Executor framework encapsulated the thread lifecycle management under interface ExecutorService. By default, the ExecutorService implementation returned by Executors needs programmer to call shutdown() method to stop it. Forget to call shutdown for an ExecutorService will prevent the JVM to exit. The JVM won't exit if there are any non-daemon thread still running. The worker threads in the ExecutorService's thread pool won't exit unless you interrupt them, in another words, they will still be running after the main thread exits. These worker threads prevent JVM to exit.

For example, the following program will never finish.

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class TheadTester {
public static void main(String... args) {
TheadTester ts = new TheadTester();

private void test() {
ExecutorService exec = Executors.newSingleThreadExecutor();
exec.execute(() -> System.out.println("ok"));

You can override the default ThreadFactory, so that the worker thread in the thread pool will be daemon thread. Since daemon thread won't prevent JVM to exit, you don't have to call ExecutorService's shutdown() method. This approach has drawbacks, though. Once the main thread exits, the only threads remain are the worker daemon threads in the threadpool. When JVM exits, these threads may still have work to do. Since they are daemons, JVM won't wait them to finish. You need to give enough time for these daemon threads to finish the work and run to the last line so that they can exit normally instead of abruptly.

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class TheadTester {
public static void main(String... args) throws InterruptedException {
TheadTester ts = new TheadTester();
//give enough time for deamon thread to finish

private void test() {
ExecutorService exec = Executors.newSingleThreadExecutor(r -> {
Thread t = new Thread(r);
return t;
exec.execute(() -> System.out.println("ok"));

This trick may be trivial, but it could be handy in big java project. For example, you need to add an ad-hoc threadpool in a class in a big project. If you use default ThreadFactory for the ExecutorService, you have to shutdown the ExecutorService somewhere. This "somewhere" might be in a thread that shutdown project level services before main thread exits, which you would rather not touch. You just want to create an ad-hoc threapool to handle events for a shot period of time, finish it, then exit. Next time, when new events come, you create a new threadpool to handle it. So on and so forth. The deamon worker strategy might be a good choice in this situation.
Continue Reading »

Tuesday, October 9, 2018

tools for handling multiple ssh on mac os x

If you ever worked in cooperate environment or be the network administrator, you sometimes need to login many servers and check stuff simultaneously.

On windows, you can use putty or smart putty to handle multiple ssh connections.

Here are some tools for mac os x equivalent.



brew install csshx


csshx --login username host1 host2 host3 host4



download the mac version from the xterm website www.iterm2.com 


add a connection profiles:

click Profiles -> Open Profiles... -> Edit Profiles -> click the + sign to add a list of hostnames and ip addresses you would like to connect to -> when creating a host record, select command radio button then type in ssh command.

open multiple profiles simultaneously in tabs:

click Profiles -> Open Profiles... -> use shift or control key to hightlight a list of hosts you want to connect to -> click New Tab
The highlighted hosts will be connected and open in tabs.

send commands to multiple tabs:

click Shell -> Broadcast Input -> Broadcast Input to All Panels in All Tabs

Continue Reading »

Monday, September 24, 2018

7 tools providing visibility for maven dependency tree

With maven pom.xml, you can have maven manage your project's dependency automatically. Whenever your project's code needs classes in a jar file, you can specify the dependency in your pom.xml, and give it a scope to specify whether this jars is needed for compile time, run time or test time. As long as the jars exist in maven repository, they will be downloaded into your local repository, which by default under ~/.m2/repository. Your project will include a set of jars which you specify in your pom.xml, they maps to the jar folders in ~/.m2/repository, and maven will include them in classpath. Those depended jars also has dependency specified in their own pom.xml. Maven goes one step further to download all the jars in the dependency chain, and have a set of rules to figure out which jar version takes precedence when it is declared in multiple places -- for example, when a jar version is declared in both parent pom and child pom, child pom takes precedence.

Enough theory about maven, here are 7 tools help you get visibility of maven dependency tree.
The example maven project used below can be checkout with "git clone https://github.com/GoogleCloudPlatform/appengine-gcs-client.git"

1. locate the dependency jar file. The groupId + artifactId + version in your pom.xml file is mapping to the corresponding directory. for example <groupId>org.codehaus.mojo</groupId> + <artifactId>versions-maven-plugin</artifactId> + <version>2.2</version> in the following example maps to ~/.m2/repository/org/codehaus/mojo/versions-maven-plugin/2.2/

>grep groupId pom.xml -A 2

>ls ~/.m2/repository/org/codehaus/mojo/versions-maven-plugin/2.2/
_remote.repositories versions-maven-plugin-2.2.jar versions-maven-plugin-2.2.pom
m2e-lastUpdated.properties versions-maven-plugin-2.2.jar.sha1 versions-maven-plugin-2.2.pom.sha1

In eclipse, you can hold command key and click a dependency in your pom.xml to step into the dependency jar's pom.xml file which located in a folder under ~/.m2/repository.

2. display dependency tree. The command "mvn dependency:tree" will display the dependency hierarchy in a tree structure. For example, com.google.oauth-client:google-oauth-client:jar:1.21.0:compile is an indirect dependency introduced by com.google.appengine.tools:appengine-gcs-client:jar:0.6:compile in your pom.xml.

>mvn dependency:tree
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building appengine-gcs-client-example 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ appengine-gcs-client-example ---
[INFO] com.google.appengine.tools:appengine-gcs-client-example:war:1.0-SNAPSHOT
[INFO] +- com.google.appengine:appengine-api-1.0-sdk:jar:1.9.36:compile
[INFO] +- com.google.appengine.tools:appengine-gcs-client:jar:0.6:compile
[INFO] |  +- com.google.guava:guava:jar:19.0-rc1:compile
[INFO] |  +- joda-time:joda-time:jar:2.3:compile
[INFO] |  +- com.google.apis:google-api-services-storage:jar:v1-rev68-1.21.0:compile
[INFO] |  |  \- com.google.api-client:google-api-client:jar:1.21.0:compile
[INFO] |  |     \- com.google.oauth-client:google-oauth-client:jar:1.21.0:compile
[INFO] |  +- com.google.api-client:google-api-client-appengine:jar:1.25.0:compile (version selected from constraint [1.19,2.0))
[INFO] |  |  +- com.google.oauth-client:google-oauth-client-appengine:jar:1.25.0:compile
[INFO] |  |  |  \- com.google.oauth-client:google-oauth-client-servlet:jar:1.25.0:compile
[INFO] |  |  |     \- com.google.http-client:google-http-client-jdo:jar:1.25.0:compile
[INFO] |  |  +- com.google.api-client:google-api-client-servlet:jar:1.25.0:compile
[INFO] |  |  |  \- javax.jdo:jdo2-api:jar:2.3-eb:compile
[INFO] |  |  |     \- javax.transaction:transaction-api:jar:1.1:compile
[INFO] |  |  \- com.google.http-client:google-http-client-appengine:jar:1.25.0:compile
[INFO] |  +- com.google.http-client:google-http-client:jar:1.25.0:compile (version selected from constraint [1.19.0,2.0))
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.5:compile
[INFO] |  |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.9:compile
[INFO] |  |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  |  \- commons-codec:commons-codec:jar:1.10:compile
[INFO] |  |  \- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] |  \- com.google.http-client:google-http-client-jackson2:jar:1.25.0:compile (version selected from constraint [1.19,2.0))
[INFO] |     \- com.fasterxml.jackson.core:jackson-core:jar:2.9.6:compile
[INFO] +- javax.servlet:servlet-api:jar:2.5:provided
[INFO] +- jstl:jstl:jar:1.2:compile
[INFO] \- com.google.appengine:appengine-api-stubs:jar:1.9.36:test
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.037 s
[INFO] Finished at: 2018-09-24T21:19:12-04:00
[INFO] Final Memory: 17M/226M
[INFO] ------------------------------------------------------------------------

In eclipse, it is equivalent to the "Effective POM" tab's output.

The eclipse even goes one step further to list all the jar dependency chains in "Dependency Hierarchy" tab. Those needed but overruled jar versions are marked as "omitted for conflict".

3. you can force maven to download the source code for your jar files if they are available in maven remote repository. "mvn dependency:sources".

You can do the same in eclipse by right click a project -> Maven -> Download Sources

4. Sometimes you get a dependent jar that almost meets your requirement except some small dissatisfaction. You can check out the source code from svn, git, etc, modify the code, then issue "mvn install". This command will publish an artifact into your local repository. Other project want to use them can reference it with a dependency entry in pom.xml. For example, run "mvn clean install" for the example will create an artifact under .m2/repository/com/google/appengine/tools/appengine-gcs-client-example/1.0-SNAPSHOT/appengine-gcs-client-example-1.0-SNAPSHOT.pom, you can reference this artifact with



in other projects' pom.xml.

You can do the same in eclipse, by right clicking the project, then selecting Run as, then clicking Maven Install. Notice due to eclipse bug,  org.codehaus.mojo:versions-maven-plugin can not be executed, I comment it out to get the maven install running. It won't hurt main functionality, this plugin as the name says, is a nice to have step during compile.

5. You can build maven project with command line, then import the built project into eclipse or have eclipse synchronize with the update. To import an maven project in eclipse, you click File -> Import -> expand maven folder, select "Existing Maven Projects" -> click Next -> browse to the pom.xml you wan to import. To update maven project in eclipse, whenever you built maven project with command line, you right click the project, select maven then click "update project...". 

6. You can inspect exactly what jars your eclipse project has in the classpath, these jars are introduced by maven plugin. 

7. Finally, you resolved the dependency issue by specifying a particular dependent jar version to override the maven chosen version or adding a list of <exclusions><exclusion></exclusion>...</exclusions>  under some <dependency></dependency> to force maven to reconsider the version. Problem now solved, you can update the project versions to a higher one to mark this fix. You can change multiple pom.xml file versions with command "mvn versions:set -DnewVersion=1.3.x-SNAPSHOT. The versions:set goal will automatically climb up local directories to find the aggregation root. It will automatically update explicitly referenced dependencies.

Continue Reading »

Wednesday, September 5, 2018

7 useful google search syntax

We use google search all the time, mostly by typing a keyword into the search bar and press enter. This gives us lots of content, sometimes more than needed. With some constrain, we can narrow down the search.
  1. "java +xyzcode" this will search java, but the page must contains xyzcode, you can change + to - to indicate that the page must not contain xyzcode.
  2. "cache:xyzcode.blogspot.com" this will search google cached page history. As a blogger, if you saved something by mistake, don't panic, google has the backup. 
  3. "site:xyzcode.blogspot.com java" this will search java only on website xyzcode.blogspot.com
  4. "filetype:pom JGraphT" this will search JGraphT only in file type pom.
  5. "inurl:xyzcode" this will search xyzcode in the url itself.
  6. "xyzcode@facebook" this will search xyzcode in facebook.
  7. "#xyzcode" this will search hashtag xyzcode.
Continue Reading »

Sunday, September 2, 2018

how to setup ASUS HOME SERVER with mac os x

This is a small trick but will be handy if you need to access the ASUS HOME SERVER with system other than windows. The menu asked the user to install an windows application in order to access the server. However, notice the ASUS HOME SERVER are actually a reconfigured windows server 2003 with IIS server 6 up running. You can login it from you LAN with microsoft remote desktop.

Download and install microsoft remote desktop, configure the connection. You can find the ASUS HOME SERVER's LAN ip from your gateway router. The server generally don't response to LAN broadcast.

connect to windows server
connect to windows server

Double click the remote desktop icon to login, when ask for the administrator password, put the one you used to setup the server at the first place.

Once logged in, the windows home server console application is right there on the desktop, double click it, then you can configure the server as if you installed a windows access app. You will feel happy because this is actually a IIS server machine with windows server 2003 functionalities.

windows home server console
windows home server console

Continue Reading »