Tuesday, March 26, 2019

2 shortcuts that saves time

In finder window, the following shortcut will allow you to go to a directory by typing a path
command + shift + g

In eclipse, the following shortcut will allow you to search a class's methods by typing the name
command + o





Wednesday, February 20, 2019

7 shell commands for network trouble-shooting

1. nslookup <url>
check what's your url is known by the dns server.

2. ping <IP>
Is the IP alive? If ping get 100% loss, it could mean two things, either the host is down or it is blocking the ICMP traffic.

3. curl <url>
check if the web application you are trying to access is alive.

4. netstat -nultp
ssh into the server and check what's going on there. run the command on the server machine to list all the listening ports of your application, for example, your java program's listening ports are:

netstat -nultp | grep java

5. nc <IP> <port>
Your web application might be listening on a port such as 8799, which you don't know how to communicate with. Use netcat, the general purpose net client to connect to the port to tell if the port is open.

6. openssl s_client -connect <IP:port>
Whenever your web application get authentication issue during network connection, get the public key from the destination server, then check whatever your web client was looking for are there.

7. tcpdump -i eth0 -s0 -v port 80
Still have issue? Try to get the pcap from the source host, load balancer, proxy server, destination host, then look into them, looking for issues such as connection reset, socket reuse, traffic drop, wait timed out, concurrency bug, etc.


Friday, February 15, 2019

7 techniques that help system administrator to find bottleneck

Your web application runs slow and your clients are complaining about long response time.

You need to figure out what's the bottleneck of your program.

  • is it cpu bound?
  • is it memory bound?
  • is it disk space bound?
  • is it network io bound?
Here are 7 techniques to help you find out.

1. shell command "uptime"


demo>uptime
18:30  up 24 days, 12:18, 3 users, load averages: 1.66 2.31 7.74


in the output, the most important information is the load averages: it is the 1 minute, 5 minute and 15 minute load average respectively. This number need to be make sense together with your cpu number.

You can find your cpu number with command:
grep -c '^processor' /proc/cpuinfo

For example, you 15 minutes load average is 7.74 and you have 2 cores. Then each core have roughly 4 processes running on it. This is far overloaded. However, if you have 8 cores, each process get one core, your cpu is running at the sweet spot. If you have 16 cores, there are about 8 cores idle, you didn't utilize your cpu resources cost-effectively.

the load average can also be found from command
top

2. shell command "free -m" 

free -m total used free shared buff/cache availableMem: 1695 517 249 0 927 1010Swap: 0 0 0
the output gives you an estimation of the memory usage in M. In the above example, you have 1010M RAM available, which is 2 times more than the amount used. Your server is not memory bound.

Again, this information can be found from the command
top

3. shell command "df -h"
df -hFilesystem Size Used Avail Use% Mounted onoverlay 36G 25G 11G 71% /tmpfs 64M 0 64M 0% /devtmpfs 848M 0 848M 0% /sys/fs/cgroup/dev/sdb1 4.8G 11M 4.6G 1% /home/dev/sda1 36G 25G 11G 71% /rootoverlayfs 1.0M 124K 900K 13% /etc/ssh/keystmpfs 848M 736K 847M 1% /run/metricsshm 64M 0 64M 0% /dev/shmoverlayfs 1.0M 124K 900K 13% /etc/ssh/ssh_host_rsa_keytmpfs 848M 0 848M 0% /run/google/devshell
The output of df let you know your disk space usage. 
In the above example, the first row tells you this: you have a 36G storage from a file system named "overlay", it is mounted on dirctory /. 71% of the disk space on that file system has been used. If you want to clean up some space there, you can 
cd /
then find the largest files occupying the space then decide which ones you want to rm.
du -h . | sort -nr | head

Similarly, the 5th row tells you: you have another 36G storage from a file system named /dev/sda1, it is mounted on directory /root. 71% of the space on that file system has been used. If you want to clean up some space there, you can 

cd /root
then find the largest files occupying the space then decide which ones you want to rm.
du -h . | sort -nr | head

4. shell command "sar -u" and "iostat -x 3"
these commands will gives you historical/realtime information about cpu/memory/disk usage, which further helps you look into hardware resource related bottleneck.

For example, if your web application is running in a VM, that VM might be hosted in a hypervisor. You have to consider the hypervisor when you analyze performance. Your hypervisor could be overloaded if the administrator put tons of VMs on one machine, or that guy simply put the hypervisor in power save mode to save bucks. You can find the tell-tell sign such as abnormal %iowait in "sar -u" output. Once you found something smelly, you can have the VM administrators to check the VM stats such as READY and Co-Stop for signs of overloading.

5. shell command "ps -ef | grep java"
the output could gives you the resource usage of your java web application, or even better, the JVM parameters of your java program. Look for parameter such as
  • -server         optimized for server application
  • -Xms           minimum heap size
  • -Xmx           maximum heap size
  • -XX:NewRatio                     ratio of young and old generation section
  • -XX:+UseG1GC                  garbage collection strategy
  • -XX:-DisableExplicitGC     prevent stop of the world full garbage collection
6. check the web container resource usage
For example, you might have tomcat as your web container. Your web container's thread pool size might be too large or too small. The web container could either wasted too much memory with a over-sized thread pool, or have the requests queued up with a too small thread pool.

7. check the web application resource usage
Have your application log the performance matrix such gc stats, cpu stats, thread pool size, database connection pool size, queue size, etc. then seek optimization opportunities. 
For example:
  • thread pool size should be decided according to the requests per second and the response time. For example, your host gets 10 requests per second, each requests need average 2 seconds to serve. You should have at least 20 threads if you don't want arriving requests to pile up waiting for service. It is a rule of thumb, 2 threads will be too few, 200 threads will be too many. You benchmark and test in order to find the number between 2 and 200.
  • if your gc happened too often or took too long, you need a bigger heap size or change the garbage collecting strategy.
  • if your threads spend most of time waiting for IO, increasing thread pool size or increasing heap size won't help. 
  • if your database query took too long, you either need a bigger database pool or explore more parallelization to fetch from database earlier, or you need to cache some contents local.
  • if your threads spends most of time idle, you want to shrink the thread pool size or increase supply. For example, giving a bigger job queue, so that there are always job available to fetch even though the requests arriving rates fluctuate. Notice an over-sized job queue could boost performance on a single host but harm overall performance in a host farm. Smaller job queue encourage more fair traffic distribution among the hosts, while larger job queue could cause one host to be overloaded, by chance, when there are spikes in the traffic.
  • if your threads spent time do task in serial, such as request another network service one by one, then find an opportunity for parallelization. Create a dedicated thread pool just to change that serial computation into a parallel one. 
  • if two computation steps don't have to be coupled, decouple them with a concurrent queue, then have a producer thread pool to feed the queue then have a consumer thread pool to consume the queue.

Thursday, February 7, 2019

How to consuming a soap endpoint in java

A soap end point looks something like
http://xyzcode.xyz/quote
usually the wsdl file can be download from
http://xyzcode.xyz/quote?wsdl

The wsdl file will give the instruction about how to consume the soap web service, that is all you needed to know in order to talk to soap api besides the user/pass for authentication.

Start with a wsdl file such as quote.wsdl, you can import the file into third-party gui tools such as soapUI, then you fill up all the fields for the request in xml format, and press send then wait, the soapUI will send your request to the remote endpoint with soap protocol, then get the response, then print the output in xml format.

Instead of manually filling the request fields in SoapUI GUI, you can have a java program to automatically fill all the needed fields, construct the request object, serialize it, have a SAAJ client send the request down to the wire with soap protocol, then wait for the SAAJ client to get the soap response back, then deserialize the response into response object.

The following tools and packages are what you need to make it happen.

1. apache wsdl2java.
Once you downloaded the tool from apache site, you can issue the following command to convert the wsdl file into corresponding java files.
The command is:
wsdl2java quote.wsdl

You most likely end up with 2 folder generated: com and org, within which, your wsdl generated java files reside.

An important thing to note is: the tool generated many ObjectFactory.java files for serializer and deserializer to consume. You can find the location of those OjectFactory.java by the following command:

find . -name ObjectFactory.java

the output could be something like:

./com/yourinventit/processing/android/serial/ObjectFactory.java
./com/yourinventit/processing/android/ObjectFactory.java
./org/helloexample/ObjectFactory.java

2. SAAJ client
The SOAP with Attachments API for Java or SAAJ provides a standard way to send XML documents over the Internet from the Java platform.

The maven saaj dependency is:
<dependency>
    <groupId>com.sun.xml.messaging.saaj</groupId>
    <artifactId>saaj-impl</artifactId>
    <version>1.5</version>
</dependency>

With saaj you can create soap calls like the following. In order to request, we need to create the Request message. Follow the same way we used to fill the request form in soapUI GUI, here we need to fill many fields in the request SOAPMessage object as well. We fill these fields with the help of wsdl generated classes.

SOAPConnectionFactory soapConnectionFactory = SOAPConnectionFactory.newInstance();
SOAPConnection connection = soapConnectionFactory.createConnection();
URL endpoint = new URL("http://xyzcode.xyz/quote");
MyPayload payload = business.getRequest();  //pass in from call param
SOAPMessage  message = funcCreateRequest(payload);  //use wsdl generated classes
SOAPMessage response = connection.call(message, endpoint);
connection.close();

3. jaxb
We said that we fill soap request fields with the help of wsdl generated classes. In fact, there are java library to help serialize and deserialize your soap request and response. The tool for that is jaxb.
You can include jaxb using maven.

<dependency>
    <groupId>javax.xml.bind</groupId>
    <artifactId>jaxb-api</artifactId>
    <version>2.2.12</version>
</dependency>
<dependency>
    <groupId>com.sun.xml.bind</groupId>
    <artifactId>jaxb-impl</artifactId>
    <version>2.2.11</version>
</dependency>

Once you have jaxb, you need to tell JAXBContext where to find the ObjectFactory.java files in order to serialize and deserialize soap objects generated by the wsdl file.
With the above example, your initialization will looks like:

JAXBContext jc = JAXBContext.newInstance("com.yourinventit.processing.android.serial:com.yourinventit.processing.android:org.helloexample:org.helloexample");

Once you have the JAXB initialized, you can convert between object and xml string.
For request

MyPayload payload = business.getRequest();  //pass in from call param
Marshaller m = jc.createMarshaller();
StringWriter w = new StringWriter();
xsw = XMLOutputFactory.newInstance().createXMLStreamWriter(w);
m.marshal(payload, xsw);  //consume objects specified in ObjectFactory.java
String requestString = w.getBuffer().toString();
SOAPMessage soapMessage = funcBuildSoapRequest(requestString);

For response
Node payload = soapMessage.getSOAPBody().getFirstChild();
String responseString  = funcConvertPayloadToString(payload);
Unmarshaller u = jc.createUnMarshaller();
xsr = XMLInputFactory.newInstance().createXMLStreamReader(new StringReader(responseString));
Object o = unmarshaller.unmarshal(xsr);  //consume objects specified in ObjectFactory

The "Object o" can be cast to one of the wsdl object, then you can pass the object around in your program.



How to solve git merge conflict with merge rebase

A git repository usually have a stable master branch for building the release artifacts, a development branch for collecting development checkins. The developers branch out bugfix and feature branches from develop branch.

At 1 pm, developer A branched out from develop a bugfix1 branch and developer B branched out feature1 branch from develop. At 2 pm, developer A merged bugfix1 branch back to develop. At 3 pm, developer B also try to merge feature1 branch back to develop. The merge failed, the reason is they both modified the same lines for the same file and git can not tell whose version to keep. This situation is called merge conflict.

merge conflict at node4
merge conflict at node4

To merge feature1 to develop, we use the following commands:
git checkout develop;
then
git merge feature1;

If the merge didn't success, the first thing we need to do is to issue
git merge --abort
on the develop branch to abort the merge. When you use bitbucket gui to create a pull request, the bitbucket do the "git merge --abort" for you on the develop branch automatically.

Next, we need to fix the feature1 branch so that the conflict won't be there when develop merge feature1 next time.

To do that, we can use the following commands, as we went through in the previous post:
git checkout feature1;
git merge develop;

This approach will create 3 logs in the develop branch after the merge: bugfix1 commit, feature1 commit, conflict resolving commit.

In this post, we will go through another approach, which is using:

git checkout feature1;
git rebase develop;

Git rebase command is one of the most powerful commands in Git. It has the ability to rewrite your repository's commit history, by rearranging, modifying, and even deleting commits.

rebase from node2 to node3
rebase from node2 to node3

Here the rebase means to give a new parent to the feature1 branch. This is what you do when you use git rebase to keep up with the upstream repository. The most important reason for using git rebase here is to change the starting point of your local branch feature1. It previously has the starting point 2, after rebase, it's starting point becomes 3. Since the starting point changes, all the content from node3 is applied to your local copy and the conflict will be introduced. Then you manually modify the files to solve the conflict, then you got a new version on top of the node3 version to commit.

This approach will create 2 logs after develop merge the feature1 branch: bugfix1 commit, feature1 commit.  Notice there is no conflict solve commit.

Now you have the local changes ready for feature1 branch. You have 2 options to merge the changes back to develop branch.

If you use pure command line for merge, you can merge develop local, then push to remote repo.
You can just
git checkout develop;
git pull;
git merge feature1;
git push origin develop;

If you are using bitbucket and pull request for merge. You can push the feature1 changes to remote repo, then create pull request. That way, the git bucket pull request merge will execute the following command for you on the remote repository with bitbucket's credential instead of yours.
git checkout develop;
git merge feature1;


Since the rebase changed the starting point of feature1, the remote branch will get confused, so if you are using bitbucket, and using pull request to merge feature1 to develop, then you need to use the following command to push your local changes about feature1 to remote repo.

git push -f origin feature1;

The -f flag means you acknowledged the fact that the starting point of you local branch differ from the starting point of the remote branch for feature1.

The process is demonstrated in the following demo:



===============
GIT>git clone https://github.com/tekgadg/Pong_sept_20_2D.git demorepo
Cloning into 'demorepo'...
remote: Enumerating objects: 6, done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 6
Unpacking objects: 100% (6/6), done.
Checking connectivity... done.
GIT>cd demorepo/
GIT>echo a > a.txt
GIT>echo b > b.txt
GIT>echo c > c.txt
GIT>git branch develop
GIT>git checkout develop
Switched to branch 'develop'
GIT>git config --global --edit
GIT>git status
On branch develop
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add --all
GIT>git commit -m "develop base commit"
[develop e45abe7] develop base commit
 3 files changed, 3 insertions(+)
 create mode 100644 a.txt
 create mode 100644 b.txt
 create mode 100644 c.txt
GIT>git branch feature1
GIT>git checkout -b bugfix1 develop
Switched to a new branch 'bugfix1'
GIT>echo xyzcode > a.txt
GIT>echo xyzcode >> b.txt
GIT>echo xyzcode >> c.txt
GIT>git commit -am "bugfix1 commit"
[bugfix1 8488ffa] bugfix1 commit
 3 files changed, 3 insertions(+), 1 deletion(-)
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge bugfix1
Updating e45abe7..8488ffa
Fast-forward
 a.txt | 2 +-
 b.txt | 1 +
 c.txt | 1 +
 3 files changed, 3 insertions(+), 1 deletion(-)
GIT>git checkout feature1
Switched to branch 'feature1'
GIT>echo xyznetwork >> a.txt
GIT>echo xyznetwork > b.txt
GIT>git rm c.txt
rm 'c.txt'
GIT>git status
On branch feature1
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

deleted:    c.txt

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

modified:   a.txt
modified:   b.txt

GIT>git commit -am "feature1 commit"
[feature1 ce6291a] feature1 commit
 3 files changed, 2 insertions(+), 2 deletions(-)
 delete mode 100644 c.txt
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge feature1
CONFLICT (modify/delete): c.txt deleted in feature1 and modified in HEAD. Version HEAD of c.txt left in tree.
Auto-merging b.txt
CONFLICT (content): Merge conflict in b.txt
Auto-merging a.txt
CONFLICT (content): Merge conflict in a.txt
Automatic merge failed; fix conflicts and then commit the result.
GIT>git diff
diff --cc a.txt
index ae5111a,56bdbb4..0000000
--- a/a.txt
+++ b/a.txt
@@@ -1,1 -1,2 +1,6 @@@
++<<<<<<< HEAD
 +xyzcode
++=======
+ a
+ xyznetwork
++>>>>>>> feature1
diff --cc b.txt
index 5e2bb23,c09e41c..0000000
--- a/b.txt
+++ b/b.txt
@@@ -1,2 -1,1 +1,6 @@@
++<<<<<<< HEAD
 +b
 +xyzcode
++=======
+ xyznetwork
++>>>>>>> feature1
* Unmerged path c.txt
GIT>git merge --abort
GIT>git checkout feature1
Switched to branch 'feature1'
GIT>git rebase develop
First, rewinding head to replay your work on top of it...
Applying: feature1 commit
Using index info to reconstruct a base tree...
M a.txt
M b.txt
M c.txt
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): c.txt deleted in feature1 commit and modified in HEAD. Version HEAD of c.txt left in tree.
Auto-merging b.txt
CONFLICT (content): Merge conflict in b.txt
Auto-merging a.txt
CONFLICT (content): Merge conflict in a.txt
Failed to merge in the changes.
Patch failed at 0001 feature1 commit
The copy of the patch that failed is found in:
   /Users/homenetwork/git/demorepo/.git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".

a
GIT>git diff
xyznetwork
diff --cc a.txt
index ae5111a,56bdbb4..0000000
--- a/a.txt
+++ b/a.txt
@@@ -1,1 -1,2 +1,6 @@@
++<<<<<<< HEAD
 +xyzcode
++=======
+ a
+ xyznetwork
++>>>>>>> feature1 commit
diff --cc b.txt
index 5e2bb23,c09e41c..0000000
--- a/b.txt
+++ b/b.txt
@@@ -1,2 -1,1 +1,6 @@@
++<<<<<<< HEAD
 +b
 +xyzcode
++=======
+ xyznetwork
++>>>>>>> feature1 commit
* Unmerged path c.txt
GIT>vi a.txt
GIT>vi b.txt
GIT>git diff
diff --cc a.txt
index ae5111a,56bdbb4..0000000
--- a/a.txt
+++ b/a.txt
diff --cc b.txt
index 5e2bb23,c09e41c..0000000
--- a/b.txt
+++ b/b.txt
* Unmerged path c.txt
GIT>git status
rebase in progress; onto 8488ffa
You are currently rebasing branch 'feature1' on '8488ffa'.
  (fix conflicts and then run "git rebase --continue")
  (use "git rebase --skip" to skip this patch)
  (use "git rebase --abort" to check out the original branch)

Unmerged paths:
  (use "git reset HEAD <file>..." to unstage)
  (use "git add/rm <file>..." as appropriate to mark resolution)

both modified:   a.txt
both modified:   b.txt
deleted by them: c.txt

no changes added to commit (use "git add" and/or "git commit -a")
GIT>git log
commit 8488ffa7e153fe97fb6566f94f544a91e11b926b
Author: you <you@example.com>
Date:   Thu Feb 7 06:57:35 2019 -0500

    bugfix1 commit

commit e45abe719c91e6932329eaca3885f7f75715ac96
Author: you <you@example.com>
Date:   Thu Feb 7 06:54:31 2019 -0500

    develop base commit

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git add --all
GIT>git status
rebase in progress; onto 8488ffa
You are currently rebasing branch 'feature1' on '8488ffa'.
  (all conflicts fixed: run "git rebase --continue")

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

modified:   a.txt
modified:   b.txt

GIT>ls
AndroidManifest.xml a.txt c.txt sketch.properties
Pong_sept_20_2D.pde b.txt code
GIT>git add c.txt
GIT>git status
rebase in progress; onto 8488ffa
You are currently rebasing branch 'feature1' on '8488ffa'.
  (all conflicts fixed: run "git rebase --continue")

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

modified:   a.txt
modified:   b.txt

GIT>git rebase --continue
Applying: feature1 commit
GIT>git status
On branch feature1
nothing to commit, working directory clean
GIT>ls
AndroidManifest.xml a.txt c.txt sketch.properties
Pong_sept_20_2D.pde b.txt code
GIT>git log
commit 995868efe3b50a562b4015ec9aeb89fda585de1d
Author: you <you@example.com>
Date:   Thu Feb 7 06:59:33 2019 -0500

    feature1 commit

commit 8488ffa7e153fe97fb6566f94f544a91e11b926b
Author: you <you@example.com>
Date:   Thu Feb 7 06:57:35 2019 -0500

    bugfix1 commit

commit e45abe719c91e6932329eaca3885f7f75715ac96
Author: you <you@example.com>
Date:   Thu Feb 7 06:54:31 2019 -0500

    develop base commit

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge feature1
Updating 8488ffa..995868e
Fast-forward
 a.txt | 3 ++-
 b.txt | 3 +--
 2 files changed, 3 insertions(+), 3 deletions(-)
GIT>ls
AndroidManifest.xml a.txt c.txt sketch.properties
Pong_sept_20_2D.pde b.txt code
GIT>git log
commit 995868efe3b50a562b4015ec9aeb89fda585de1d
Author: you <you@example.com>
Date:   Thu Feb 7 06:59:33 2019 -0500

    feature1 commit

commit 8488ffa7e153fe97fb6566f94f544a91e11b926b
Author: you <you@example.com>
Date:   Thu Feb 7 06:57:35 2019 -0500

    bugfix1 commit

commit e45abe719c91e6932329eaca3885f7f75715ac96
Author: you <you@example.com>
Date:   Thu Feb 7 06:54:31 2019 -0500

    develop base commit

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git status
On branch develop
nothing to commit, working directory clean
GIT>

Wednesday, February 6, 2019

How to solve merge conflict with git merge

A git repository usually have a stable master branch for building the release artifacts, a development branch for collecting development checkins. The developers branch out bugfix and feature branches from develop branch.

At 1 pm, developer A branched out from develop a bugfix1 branch and developer B branched out feature1 branch from develop. At 2 pm, developer A merged bugfix1 branch back to develop. At 3 pm, developer B also try to merge feature1 branch back to develop. The merge failed, the reason is they both modified the same lines for the same file and git can not tell whose version to keep. This situation is called merge conflict.

merge conflict at node4
merge conflict at node4


To merge feature1 to develop, we use the following commands:
git checkout develop;
then
git merge feature1;

If the merge didn't success, the first thing we need to do is to issue
git merge --abort
on the develop branch to abort the merge. When you use bitbucket gui to create a pull request, the bitbucket do the "git merge --abort" for you on the develop branch automatically.

Next, we need to fix the feature1 branch so that the conflict won't be there when develop merge feature1 next time.

To do that, we use the following commands:
git checkout feature1;
git merge develop;

This time instead of merge feature into develop, we merge develop into feature instead. As you can tell, no matter which direction the merge goes, the conflict will be there. We need to manually modify the conflicted files then commit a new version into feature1 branch. During our merge, the git knows our feature1 version is later than the develop version. So when we checkout develop then merge feature1, the feature1 version will override the develop version, so the merge will success.

Finally you use to sync your local and remote repo for the develop branch.
git push origin develop;

This approach will create 3 logs in the develop branch after the merge: bugfix1 commit, feature1 commit, conflict resolving commit.

For the developers that use bitbucket and pull request to merge feature and bugfix branches to develop. They need to solved the conflict local for feature1 branch, then
git push origin feature1;

then use pull request in bitbucket for merging into develop branch.

The process is demonstrated in the following demo:

In the next post, we will introduce an alternative approach with git rebase.
=============

GIT>git clone https://github.com/tekgadg/Pong_sept_20_2D.git demorepo
Cloning into 'demorepo'...
remote: Enumerating objects: 6, done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 6
Unpacking objects: 100% (6/6), done.
Checking connectivity... done.
GIT>cd demorepo/
GIT>git config --global user.email you@example.com
GIT>git config --global user.name "you"
GIT>git branch develop
GIT>git checkout develop
Switched to branch 'develop'
GIT>echo a > a.txt
GIT>ech b > b.txt
-bash: ech: command not found
GIT>echo b > b.txt
GIT>echo c > c.txt
GIT>git commit -am "develop base"
On branch develop
Untracked files:
a.txt
b.txt
c.txt

nothing added to commit but untracked files present
GIT>git status
On branch develop
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add .
GIT>git commit -m "develop base"
[develop 1a27b56] develop base
 3 files changed, 3 insertions(+)
 create mode 100644 a.txt
 create mode 100644 b.txt
 create mode 100644 c.txt
GIT>git branch feature1
GIT>git branch bugfix1
GIT>git branch
  bugfix1
* develop
  feature1
  master
GIT>git checkout bugfix1
Switched to branch 'bugfix1'
GIT>echo xyzcode >> a.txt
GIT>echo xyzcode >> b.txt
GIT>git rm c.txt
rm 'c.txt'
GIT>git diff
diff --git a/a.txt b/a.txt
index 7898192..8267385 100644
--- a/a.txt
+++ b/a.txt
@@ -1 +1,2 @@
 a
+xyzcode
diff --git a/b.txt b/b.txt
index 6178079..5e2bb23 100644
--- a/b.txt
+++ b/b.txt
@@ -1 +1,2 @@
 b
+xyzcode
GIT>git commit -am "bugfix1 commit"
[bugfix1 570873c] bugfix1 commit
 3 files changed, 2 insertions(+), 1 deletion(-)
 delete mode 100644 c.txt
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge bugfix1
Updating 1a27b56..570873c
Fast-forward
 a.txt | 1 +
 b.txt | 1 +
 c.txt | 1 -
 3 files changed, 2 insertions(+), 1 deletion(-)
 delete mode 100644 c.txt
GIT>git status
On branch develop
nothing to commit, working directory clean
GIT>git checkout feature1
Switched to branch 'feature1'
GIT>echo xyznetwork >> a.txt
GIT>echo xyznetwork > b.txt
GIT>echo xyznetwork >> c.txt
GIT>git commit -am "feature1 commit"
[feature1 6efd19f] feature1 commit
 3 files changed, 3 insertions(+), 1 deletion(-)
GIT>git diff
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge feature1
CONFLICT (modify/delete): c.txt deleted in HEAD and modified in feature1. Version feature1 of c.txt left in tree.
Auto-merging b.txt
CONFLICT (content): Merge conflict in b.txt
Auto-merging a.txt
CONFLICT (content): Merge conflict in a.txt
Automatic merge failed; fix conflicts and then commit the result.
GIT>git diff
diff --cc a.txt
index 8267385,56bdbb4..0000000
--- a/a.txt
+++ b/a.txt
@@@ -1,2 -1,2 +1,6 @@@
  a
++<<<<<<< HEAD
 +xyzcode
++=======
+ xyznetwork
++>>>>>>> feature1
diff --cc b.txt
index 5e2bb23,c09e41c..0000000
--- a/b.txt
+++ b/b.txt
@@@ -1,2 -1,1 +1,6 @@@
++<<<<<<< HEAD
 +b
 +xyzcode
++=======
+ xyznetwork
++>>>>>>> feature1
* Unmerged path c.txt
GIT>git merge --abort
GIT>git checkout feature1
Switched to branch 'feature1'
GIT>git checkout develop
Switched to branch 'develop'
GIT>git diff
GIT>git checkout feature1
Switched to branch 'feature1'
GIT>git merge develop
CONFLICT (modify/delete): c.txt deleted in develop and modified in HEAD. Version HEAD of c.txt left in tree.
Auto-merging b.txt
CONFLICT (content): Merge conflict in b.txt
Auto-merging a.txt
CONFLICT (content): Merge conflict in a.txt
Automatic merge failed; fix conflicts and then commit the result.
a
GIT>git diff
xyznetwork
diff --cc a.txt
index 56bdbb4,8267385..0000000
--- a/a.txt
+++ b/a.txt
@@@ -1,2 -1,2 +1,6 @@@
  a
++<<<<<<< HEAD
 +xyznetwork
++=======
+ xyzcode
++>>>>>>> develop
diff --cc b.txt
index c09e41c,5e2bb23..0000000
--- a/b.txt
+++ b/b.txt
@@@ -1,1 -1,2 +1,6 @@@
++<<<<<<< HEAD
 +xyznetwork
++=======
+ b
+ xyzcode
++>>>>>>> develop
* Unmerged path c.txt
GIT>vi a.txt
GIT>vi b.txt
GIT>git rm c.txt
c.txt: needs merge
rm 'c.txt'
GIT>git diff
diff --cc a.txt
index 56bdbb4,8267385..0000000
--- a/a.txt
+++ b/a.txt
diff --cc b.txt
index c09e41c,5e2bb23..0000000
--- a/b.txt
+++ b/b.txt
@@@ -1,1 -1,2 +1,3 @@@
 +xyznetwork
+ b
+ xyzcode
GIT>git status
On branch feature1
You have unmerged paths.
  (fix conflicts and run "git commit")

Changes to be committed:

deleted:    c.txt

Unmerged paths:
  (use "git add <file>..." to mark resolution)

both modified:   a.txt
both modified:   b.txt

GIT>git commit -am "conflict resolve with merge"
[feature1 b863a0d] conflict resolve with merge
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge feature1
Updating 570873c..b863a0d
Fast-forward
 b.txt | 1 +
 1 file changed, 1 insertion(+)
GIT>git status
On branch develop
nothing to commit, working directory clean
GIT>git branch
  bugfix1
* develop
  feature1
  master
GIT>git branch --delete bugfix1 feature1
Deleted branch bugfix1 (was 570873c).
Deleted branch feature1 (was b863a0d).
GIT>git branch
* develop
  master
GIT>git log
commit b863a0d537b6201495e4a7169883f32180b07f0e
Merge: 6efd19f 570873c
Author: you <you@example.com>
Date:   Wed Feb 6 07:44:21 2019 -0500

    conflict resolve with merge

commit 6efd19f7745211562b3ad33d079c55aae6beeb43
Author: you <you@example.com>
Date:   Wed Feb 6 07:40:53 2019 -0500

    feature1 commit

commit 570873c48d0e69c6a45e9b7d5891c7c35937c300
Author: you <you@example.com>
Date:   Wed Feb 6 07:39:04 2019 -0500

    bugfix1 commit

commit 1a27b5688d18352f747e7eea7ada7c65aeb8e90c
Author: you <you@example.com>
Date:   Wed Feb 6 07:37:08 2019 -0500

    develop base

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>

Tuesday, February 5, 2019

7 tools that monitor your servers and web applications

Your web application is a distributed software system that runs 24/7 on a fleets of servers. There are many functional components such as DNS server, load balancer, router, web server, authentication service, provisioning service, redex server, database server etc. Constantly they are generating data such as request count, response time, cpu load, ram usage, disk usage, web container thread pool size, application thread pool size, database connection pool size, garbage collection time, etc. For each data source, you can get static data such as average, sum, min, max, perc99, perc95, etc. You can then sort them by site, hostname, IP, date, tag, source etc. Sometimes, there are correlations between data types. For example high request count often correlates to high cpu load, high ram, high thread-pool size and longer database response time, etc.

The following 7 tools will help you to collect the data from your application and organize them in smart ways, so that your machines are under control.

1. Event queues

Your web application is a group of applications. Each application is deployed on a fleet of servers behind load balancer, and each copy of the artifact deployed on each server is a multi-threading program. As you can tell, there are high chance for race conditions. We don't want our application meta data collection ruin the main functionality or create synchronize bottleneck. A thread-safe queue implementation can help a lot in this situation. It decouples producer and consumer, so that the log generated from one thread in one VM won't mess up with another thread in the same VM or another VM. For example Kafka and RabbitMQ are two of this kind.  Your application can send events as object to a topic/stream of the queue. These topics, after processing, can be sent to the API of consumers such as another app, database, zabbix, splunk, tableau etc.

2. Data analytic tools

Your Kafka stream can be sent to data analytic tools such as splunk. Once the data get there, they are stored in the way fast retrieving and sorting are possible. Splunk can analyze huge amount of data, use them to create statistics, time charts and dashboards. Splunk also allows events to trigger alerts. The alert threshold criteria can range from count larger than a number to time longer than a value. Once the alert is triggered, actions can be taken, for example, send message to slack channel, email, SMS, ticketing API.

3. Incident Response Platform

There are many ticketing platforms such as bmc remedy, salesforce, pagerduty. Pagerduty for example, allows you to install an app in splunk. One of the splunk alert action is send to a pagerduty service. Once the pagerduty service receives a call, it acts according to the escalation policy -- for example, call the primary contacts, if not get acknowledgement within 5 minutes, call the secondary contacts, so on and so forth, until someone takes action about the event-- could be annoying during night hours.

4. Host inventory tools

Your server warehouse need management. Hardware as well as VMs can be managed with tools such as zabbix. Once the zabbix agent is installed on an unix server, it starts to collect infrastructure informations such as cpu, memory, hdc io, hdc bw, sda io, sda bw, etc. It allows monitor, inventory and report about infrastructure hosts on the Zabbix server. You can also draw graph with your hardware resource usage history.

5. VM and cloud inventory tools

Virtual machines and cloud VM instances are special resources. They are special because they are elastic and volatile. Tools such as VMWare Vcenter/Vsphere, AWS, GCP, AZURE allows you to quickly create/destroy VMs, create cloud virtual company, reconfigure routing rule, setup ACL, configure scale up/down policy etc. Oracle has its own JVM debug tool which can be invoked with command JVisualVM.

6. Data warehouse mining and visualization tools

Tools such as tableau can connect to almost any data warehouse applications on the market: oracle, mysql, AWS Redshift, cubes, Teradata, cassandra, data lake, redis, microsoft SQL Server, mongodb, hyperion, etc. With the wealth of data already stored in the data warehouse, tableau can generate reports, statistics, graphs across different data sources and give the user power to further analyze them.

7. Network inventory tool

Your network resources such as IPs, Nodes, Servers need to be organized and grouped. There are tools such as NGINX, BIG IP to help you out. At anytime, you are able to inventory your network resource by locations, OS, liveness, cells etc. You are also able to take a group of servers out of service or put them into service.

7 git undo commands you should know

In git, you can create new files then add them into git staging area, then commit them into local repo history. At any time, you can undo those git adds and commits.

There are a group of commands that allow you undo:
  1. rm -rf {filename}
  2. git reset HEAD
  3. git reset HEAD {filename}
  4. git reset --hard HEAD
  5. git reset --hard origin/{branchname}
  6. git checkout -f
  7. git revert HEAD
First of all untracked files should not be the git's concern, unless you use rm {filename} to remove them from your disk, any commands start with git (except git clean) should not touch them.

git reset HEAD
this command put the files in the staging area back to the untracked state.

git checkout -f
this command will replace the local files with the remote version. Of course, your untracked files will remain untouched. It is more destructive than git reset HEAD, since the files in staging area are wiped out from disk, they are not put back into untracked state.

notice Neither "git checkout -f", "git reset HEAD" nor "git reset --hard HEAD" will touch the committed files.
In order to undo commit, we need to issue command "git reset --hard orign/{branchname}". This command is destructive -- it won't put the committed files back to untracked the state, it wipes them away from the disk.

Another way to undo commit is to use
git revert HEAD
it added a new commit to undo the local commits so that the end result is the same of HEAD version. It is a safer way of undo commit since it kept a copy of you committed files in history.

exmple
============

GIT>git clone https://github.com/tekgadg/Pong_sept_20_2D.git demorepo
Cloning into 'demorepo'...
remote: Enumerating objects: 6, done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 6
Unpacking objects: 100% (6/6), done.
Checking connectivity... done.
GIT>cd demorepo/
GIT>ls
AndroidManifest.xml code
Pong_sept_20_2D.pde sketch.properties
GIT>echo a > a.txt
GIT>echo b > b.txt
GIT>echo c > c.txt
GIT>mkdir test
GIT>echo d > test/d.txt
GIT>echo e > test/e.txt
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add --all
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   a.txt
new file:   b.txt
new file:   c.txt
new file:   test/d.txt
new file:   test/e.txt

GIT>git reset HEAD test/
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   a.txt
new file:   b.txt
new file:   c.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

test/

GIT>git checkout -f
Your branch is up-to-date with 'origin/master'.
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>ls
AndroidManifest.xml code test
Pong_sept_20_2D.pde sketch.properties
GIT>echo a > a.txt
GIT>echo b > b.txt
GIT>echo c > c.txt
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add -u .
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add .
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   a.txt
new file:   b.txt
new file:   c.txt
new file:   test/d.txt
new file:   test/e.txt

GIT>git reset HEAD c.txt test
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   a.txt
new file:   b.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

c.txt
test/

GIT>git commit -m "step 1 commit 2 files"
[master 6ff2dba] step 1 commit 2 files
 Committer: Home Network <homenetwork@OK-MBP.home>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:

    git config --global --edit

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 2 files changed, 2 insertions(+)
 create mode 100644 a.txt
 create mode 100644 b.txt
GIT>git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
Untracked files:
  (use "git add <file>..." to include in what will be committed)

c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git log
commit 6ff2dba507fd244988282078e349f34807ca9bb9
Author: Home Network <homenetwork@OK-MBP.home>
Date:   Tue Feb 5 08:14:49 2019 -0500

    step 1 commit 2 files

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git reset HEAD
GIT>git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
Untracked files:
  (use "git add <file>..." to include in what will be committed)

c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git log
commit 6ff2dba507fd244988282078e349f34807ca9bb9
Author: Home Network <homenetwork@OK-MBP.home>
Date:   Tue Feb 5 08:14:49 2019 -0500

    step 1 commit 2 files

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git reset --hard HEAD
HEAD is now at 6ff2dba step 1 commit 2 files
GIT>git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
Untracked files:
  (use "git add <file>..." to include in what will be committed)

c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git log
commit 6ff2dba507fd244988282078e349f34807ca9bb9
Author: Home Network <homenetwork@OK-MBP.home>
Date:   Tue Feb 5 08:14:49 2019 -0500

    step 1 commit 2 files

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git reset --hard origin/master
HEAD is now at 40aac7d upload
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git log
commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>ls
AndroidManifest.xml c.txt sketch.properties
Pong_sept_20_2D.pde code test
GIT>echo a > a.txt
GIT>echo b > b.txt
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add test/
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   test/d.txt
new file:   test/e.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt

GIT>git checkout -f
Your branch is up-to-date with 'origin/master'.
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt

nothing added to commit but untracked files present (use "git add" to track)
GIT>ls
AndroidManifest.xml b.txt sketch.properties
Pong_sept_20_2D.pde c.txt
a.txt code
GIT>git add c.txt
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   c.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt


Wednesday, January 30, 2019

7 Software release tools you should know

DevOps
DevOps


You can do your release management as simple as command-line: mvn release:prepare;mvn release:perform.

However, there are lots of tools that make your release great and pleasant.
1. jira
Your project manager and scrum master can manage the release process with jira. A jira can integrate with bitbucket, slack, confluence etc. You can branch out github code base or create pull request in jira. Whenever you checked in code or created pull request, the jira will show updates; your co-worker can even get email when their name get mentioned or pull request is assigned to them. You can created all sorts of jiras: story, task, deployment, bug, feature, incident, etc. These jiras can have all sorts of relationships: subtask, related to, block, epics, etc. Isn't that pleasant?

2. bitbucket, github
You can do all git action with command line, however git management tools such as bitbucket makes your git usage much more efficient. In bitbucket, you can do basic operations such as create repository, create branches, create pull request, code review, merge, etc. It can do more than that. When bitbucket is integrated with bamboo or jenkins, your code check-in will trigger auto-build, and the build result can be a factor in the merge criteria. Besides, your merge will trigger auto version update, isn't it magical? Whenever you want, you can get lots of information by searching code base, lookup pull requests, filter on pull requests, compare tagged versions, and lots more.

3. sonatype nexus
Maven central repository allow your maven project to find dependencies. However, if you want to manage your own private jar collections, your may need a repository manager server like sonatype nexus. It allows you to proxy, collect, and manage your dependencies. With a few lines about the repo server url and credential added into ~/.m2/settings.xml file, the developers can publish to your sonatype nexus server hosted repos, this will also allow your mvn to use your repo as a mirror of central.

4. IC tools
Tools such teamcity, jenkins, electric cloud, bamboo act as the continuous integration build server. These build servers tie automated builds, tests, and releases together in a single workflow. For example, when bamboo is integrated with bitbucket, code check in will trigger bamboo builds. Some builds create snapshot artifacts, others create released artifacts. With the gui, you can create new build project, set up the trigger criteria, specify the git repo, specify the test/release command such as
mvn release:prepare;mvn release:perform;
You can even setup the quality gates by adding Nexus IQ and SonarQube etc. as release gate tasks.

5. Nexus IQ
You don't want to have code vulnerability to be in production, so you can setup Nexus IQ tool to scan and find bad codes. The Nexus IQ setup gui allows you to customize policy, so you can define which components are acceptable, and which are not.

6. SonarQube
Besides code coverage, SonarQube can do more such as finding smelly code or bad style, so your code is not only reliable but also smells good.

7. Automatic deployment tools
After IC tools finished test and release, they will create artifacts that need to be deployed on various servers on your DEV, QA, PILOT or PROD environment. You can use scp commands to copy the artifact such as jar, war, zip, tar.gz files on the target server's web-service (such as tomcat/jboss) directory. However, you don't have to do this kind of routine manually. Your IC tool can trigger an automatic deployment job as one of the release task. For example, automatic deployment tools such as RunDeck, Puppet, bladelogic etc. can execute user defined deployment jobs on server nodes. These nodes are your unix servers located on different environment. The artifacts are push to those nodes automatically at the end of the IC process.




Thursday, January 24, 2019

maven trouble-shooting checklist

In theory, in order to use maven to build a project, all you need to do is to download maven, then update ~/.bashrc to have maven bin in $PATH.

export PATH=/Users/youruser/Downloads/apache-maven-3.2.5/bin/:$PATH

However, in reality, sometimes your maven gets stuck when downloading dependencies. Don't blame yourself for fat fingers, for big projects, the maven build process different every single time! First of all, your dependency could have snapshot versions, which might get updated on the repo 5 minutes ago, so when maven downloaded the latest one, trouble is introduced. Even though your direct dependency have no snapshot version. Your indirect dependency could have snapshot version dependency, downloading them also brings new jars which brings trouble into your build process. Besides, your OS/network admin could applied rules on your account, so that some of your remote repos can no longer be reached. Your build will still go on because your cached jar in local repo, however the cached version is now differ from the latest version, so your build process will differ from your coworkers' build process. Even though nothing happens in the environment, your download could be erroneous during file transfer due to network package loss, memory buffer, stale file stub etc.

Here are a trouble-shooting checklist.

1. find which maven is used.

which mvn

This command also tells you the maven home location.

There are two locations where a settings.xml file may live:
The Maven install: ${maven.home}/conf/settings.xml
A user’s install: ~/.m2/settings.xml
The user's install takes precedence than the other.

check both settings.xml file to make sure what properties are override. Without override, you default maven repo should locate inside ~/.m2/repository

2.  find if the complained jar file is in your local and remote repo or not.
The maven build will check local cache first, then go to download the remote repo for snapshot versions and won't download the release versions.
If the jar file is in the local repository ~/.m2/repository. It could be corrupted during download. You can use
mvn clean -U package
to force maven to re-download the existing local repository files from remote repository.
To achieve the same goal, you can also delete that dependency folder, or even the entire ~/.m2/repository folder, so that maven can not find the jar in local cache and have to trip to the remote repository to re-download everything.

Notice often times, the maven is complain about a jar and its depended jar file chains -- check each one of them, not just the first one in the complained list!

If you are helping others to trouble shoot build issue and the complained jar files are in your local repo but not in the remote repo. Chances are it just builds for you but won't build for others. Because someone has deleted the needed jar files in the remote repository, leaving a bug in the unaware victim code base.

3. restart you computer if possible. It will fix weird issues related to memory and network policy etc. For example network policy changes when you work from home with vpn then work in office.

4. check eclipse maven settings.
in eclipse -> preferences -> maven -> user settings, make sure you didn't override any default settings, so that you are using the default local repository. If you don't want to rely on default, literally specify the default location so that you are sure.

right click the project, then "maven", then "update project..", you can check the force update check box to have eclipse do the equivalent of
mvn clean -U package

5. restart eclipse, eclipse could run into weird issue internally.

6. ask if other people has the same build issue, then ask them to send you the ~/.bashrc, ~/.bash_profile, ~/.m2/settings.xml, and the /path-to-maven-home/conf/settings.xml. Also ask for the result of ls -lrt ~/.m2/repository/path/to/complained/jar

7. if you are using git or svn, find out who added the dependency that causes the trouble, have a conversation about it.

svn to git migration

Migrating svn to git needs some plan. There is a period, when the new git repo is created but the developers are continue to commit to the old svn repo.

Bitbucket provided a tool called svn-migration-scripts.jar to make the process easy. Down load the jar file and put under home directory ~/.

5 steps migration.


step 1. create a case-insensitive file-system image. 

If you are using linux such as redhat, ubuntu, your system already case-sensitive. If you are using mac os X, your system is using case-insensitive file-system. Run the following command to tell:

java -jar ~/svn-migration-scripts.jar verify

If the output looks like this, you then need to create a image.
"You appear to be running on a case-insensitive file-system. This is unsupported, and can result in data loss."

java -jar ~/svn-migration-scripts.jar create-disk-image 5 GitMigration

once the directory is created, cd there

cd ~/GitMigration

step 2. Extract author information

java -jar ~/svn-migration-scripts.jar authors https://svn.example.com > authors.txt

modify the emails in authors.txt

step 3. clone the svn to git repo

if your svn don't have a trunk, branch, tag lay out, use the following command.
git svn clone -authors-file=authors.txt https://svn.yoursvnrepo.com/YourSvnProj YourSvnProjAsGit

otherwise, you can specify the location of your trunk, branches, tag, use the following command instead
git svn clone --trunk=/trunk --branches=/branches --branches=/bugfixes --tags=/tags --authors-file=authors.txt https://svn.yoursvnrepo.com/YourSvnProj YourSvnProjAsGit

now clean the created project.
java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git --force

step 4. upload to git repo

Once you created a new git repo in bitbucket, you can push your local repo to the remote repo.

git remote add origin https://<user>@bitbucket.org/<user>/<repo>.git
git push -u origin --all
git push --tags

step 5. start to use git repo

now your coworker can start to clone the git repo and work on it.
git clone https://<user>@bitbucket.org/<user>/<project>.git <destination>

for the users that are still committing to the old svn, use the following command to synchronize

git config svn.authorsfile <path-to-authors-file>
git svn fetch
java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar sync-rebase
java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git --force

This Ocean Waves Power Plant is The First of Its Kind In All of Latin Am...

Thursday, January 10, 2019

AI found Mysterious fast radio bursts from deep space, ‘could be aliens’

Canadian-led team of astronomers searching for FRBs have recently detected a repeating FRBs that could be alien signal. With the Breakthrough Listen program used AI (artificial intelligence), they have found more than 60 FRBs, two of the FRBs was detected repeating.

The origin of fast radio bursts (FRBs), millisecond-long pulses of radio waves, may be generated by black holes or super-dense neutron stars merging together. However, for such astrophysical phenomena events randomly repeated at the same location twice has a tiny chance.



search keywords: FRBs, alien

First Commercial Quantum Computing System from IBM

IBM debuts the Q System One, an integrated quantum computing system made for commercial and research use, the first of its kind in the world.

The system can be used to predict events that are almost impossible for classical computing, but which quantum computing is uniquely adept at solving. For example, predict financial events, optimize logistic problems.

𝑖Ψ(𝑞,𝑡)𝑡=HΨ(𝑞,𝑡)
Quantum mechanics describe our world as entangled waves in time and space (or a set of vectors where time and space dimensions are just mathematical presentations), instead of collection of small particles moving in space. This means that, because of this property of measurement in quantum mechanics, our universe is inherently a non-local, and everything seems connected. In another word, any quantum mechanics interaction has the whole universe considered, that's why it is simpler to predict financial events for quantum mechanics -- complexity and considering the influence of outside world is not a problem at all in quantum mechanics!


Here is an example program I ran on a 3 qubits IBM Q quantum computer. It demonstrate the quantum entanglement.

entanglement demonstration on IBM Q quantum computer
Entanglement is the weirdest of all quantum phenomena. Two or more quantum objects are entangled when, despite being too far apart to influence one another, they behave in ways that are 1) individually random, but also 2) too strongly correlated to be explained by supposing that each object is independent from the other. As the basic phenomena of physics, entanglement is everywhere. Some particles or waves in your body and my body might be in entangled states now, they go hand in hand. It is mind blow -- when your brain managed to flip one wave in your body, it flipped the other half in mine too, even though I never know you. Yes, no matter who we are, our thought litterally changed the world.
See the demo result? We only got 11 and 00 states, there is no 10 and 01 states. The 2 particles always agree with each other. In theory, the two qubits always agree, even you move one of them zillions of miles away.
include "qelib1.inc";
qreg q[2];
creg c[2];
h q[0];
cx q[0],q[1];
measure q[0] -> c[0];
measure q[1] -> c[1];

entanglement demonstration
entanglement demonstration

search keyword: IBM CES 2019 First Integrated Quantum Computing System, IBM Q

chatgpt's view about universe

 is there god? The existence of a higher power, commonly referred to as "God," is a matter of personal belief and faith. There is ...