7 shell commands for network trouble-shooting

1. nslookup <url>
check what's your url is known by the dns server.

2. ping <IP>
Is the IP alive? If ping get 100% loss, it could mean two things, either the host is down or it is blocking the ICMP traffic.

3. curl <url>
check if the web application you are trying to access is alive.

4. netstat -nultp
ssh into the server and check what's going on there. run the command on the server machine to list all the listening ports of your application, for example, your java program's listening ports are:

netstat -nultp | grep java

5. nc <IP> <port>
Your web application might be listening on a port such as 8799, which you don't know how to communicate with. Use netcat, the general purpose net client to connect to the port to tell if the port is open.

6. openssl s_client -connect <IP:port>
Whenever your web application get authentication issue during network connection, get the public key from the destination server, then check whatever your web client was looking for are there.

7. tcpdump -i eth0 -s0 -v port 80
Still have issue? Try to get the pcap from the source host, load balancer, proxy server, destination host, then look into them, looking for issues such as connection reset, socket reuse, traffic drop, wait timed out, concurrency bug, etc.

7 techniques that help system administrator to find bottleneck

Your web application runs slow and your clients are complaining about long response time.

You need to figure out what's the bottleneck of your program.

• is it cpu bound?
• is it memory bound?
• is it disk space bound?
• is it network io bound?

Here are 7 techniques to help you find out.

1. shell command "uptime"

demo>uptime

18:30  up 24 days, 12:18, 3 users, load averages: 1.66 2.31 7.74

in the output, the most important information is the load averages: it is the 1 minute, 5 minute and 15 minute load average respectively. This number need to be make sense together with your cpu number.

You can find your cpu number with command:

grep -c '^processor' /proc/cpuinfo

For example, you 15 minutes load average is 7.74 and you have 2 cores. Then each core have roughly 4 processes running on it. This is far overloaded. However, if you have 8 cores, each process get one core, your cpu is running at the sweet spot. If you have 16 cores, there are about 8 cores idle, you didn't utilize your cpu resources cost-effectively.

the load average can also be found from command

top

2. shell command "free -m" 

free -m total used free shared buff/cache availableMem: 1695 517 249 0 927 1010Swap: 0 0 0

the output gives you an estimation of the memory usage in M. In the above example, you have 1010M RAM available, which is 2 times more than the amount used. Your server is not memory bound.

Again, this information can be found from the command

top

3. shell command "df -h"

df -hFilesystem Size Used Avail Use% Mounted onoverlay 36G 25G 11G 71% /tmpfs 64M 0 64M 0% /devtmpfs 848M 0 848M 0% /sys/fs/cgroup/dev/sdb1 4.8G 11M 4.6G 1% /home/dev/sda1 36G 25G 11G 71% /rootoverlayfs 1.0M 124K 900K 13% /etc/ssh/keystmpfs 848M 736K 847M 1% /run/metricsshm 64M 0 64M 0% /dev/shmoverlayfs 1.0M 124K 900K 13% /etc/ssh/ssh_host_rsa_keytmpfs 848M 0 848M 0% /run/google/devshell

The output of df let you know your disk space usage. 

In the above example, the first row tells you this: you have a 36G storage from a file system named "overlay", it is mounted on dirctory /. 71% of the disk space on that file system has been used. If you want to clean up some space there, you can 

cd /

then find the largest files occupying the space then decide which ones you want to rm.

du -h . | sort -nr | head

Similarly, the 5th row tells you: you have another 36G storage from a file system named /dev/sda1, it is mounted on directory /root. 71% of the space on that file system has been used. If you want to clean up some space there, you can 

cd /root

then find the largest files occupying the space then decide which ones you want to rm.

du -h . | sort -nr | head

4. shell command "sar -u" and "iostat -x 3"

these commands will gives you historical/realtime information about cpu/memory/disk usage, which further helps you look into hardware resource related bottleneck.

For example, if your web application is running in a VM, that VM might be hosted in a hypervisor. You have to consider the hypervisor when you analyze performance. Your hypervisor could be overloaded if the administrator put tons of VMs on one machine, or that guy simply put the hypervisor in power save mode to save bucks. You can find the tell-tell sign such as abnormal %iowait in "sar -u" output. Once you found something smelly, you can have the VM administrators to check the VM stats such as READY and Co-Stop for signs of overloading.

5. shell command "ps -ef | grep java"

the output could gives you the resource usage of your java web application, or even better, the JVM parameters of your java program. Look for parameter such as

• -server         optimized for server application
• -Xms           minimum heap size
• -Xmx           maximum heap size
• -XX:NewRatio                     ratio of young and old generation section
• -XX:+UseG1GC                  garbage collection strategy
• -XX:-DisableExplicitGC     prevent stop of the world full garbage collection

6. check the web container resource usage

For example, you might have tomcat as your web container. Your web container's thread pool size might be too large or too small. The web container could either wasted too much memory with a over-sized thread pool, or have the requests queued up with a too small thread pool.

7. check the web application resource usage

Have your application log the performance matrix such gc stats, cpu stats, thread pool size, database connection pool size, queue size, etc. then seek optimization opportunities. 

For example:

• thread pool size should be decided according to the requests per second and the response time. For example, your host gets 10 requests per second, each requests need average 2 seconds to serve. You should have at least 20 threads if you don't want arriving requests to pile up waiting for service. It is a rule of thumb, 2 threads will be too few, 200 threads will be too many. You benchmark and test in order to find the number between 2 and 200.
• if your gc happened too often or took too long, you need a bigger heap size or change the garbage collecting strategy.
• if your threads spend most of time waiting for IO, increasing thread pool size or increasing heap size won't help. 
• if your database query took too long, you either need a bigger database pool or explore more parallelization to fetch from database earlier, or you need to cache some contents local.
• if your threads spends most of time idle, you want to shrink the thread pool size or increase supply. For example, giving a bigger job queue, so that there are always job available to fetch even though the requests arriving rates fluctuate. Notice an over-sized job queue could boost performance on a single host but harm overall performance in a host farm. Smaller job queue encourage more fair traffic distribution among the hosts, while larger job queue could cause one host to be overloaded, by chance, when there are spikes in the traffic.
• if your threads spent time do task in serial, such as request another network service one by one, then find an opportunity for parallelization. Create a dedicated thread pool just to change that serial computation into a parallel one. 
• if two computation steps don't have to be coupled, decouple them with a concurrent queue, then have a producer thread pool to feed the queue then have a consumer thread pool to consume the queue.

How to consuming a soap endpoint in java

A soap end point looks something like
http://xyzcode.xyz/quote
usually the wsdl file can be download from
http://xyzcode.xyz/quote?wsdl

The wsdl file will give the instruction about how to consume the soap web service, that is all you needed to know in order to talk to soap api besides the user/pass for authentication.

Start with a wsdl file such as quote.wsdl, you can import the file into third-party gui tools such as soapUI, then you fill up all the fields for the request in xml format, and press send then wait, the soapUI will send your request to the remote endpoint with soap protocol, then get the response, then print the output in xml format.

Instead of manually filling the request fields in SoapUI GUI, you can have a java program to automatically fill all the needed fields, construct the request object, serialize it, have a SAAJ client send the request down to the wire with soap protocol, then wait for the SAAJ client to get the soap response back, then deserialize the response into response object.

The following tools and packages are what you need to make it happen.

1. apache wsdl2java.
Once you downloaded the tool from apache site, you can issue the following command to convert the wsdl file into corresponding java files.
The command is:
wsdl2java quote.wsdl

You most likely end up with 2 folder generated: com and org, within which, your wsdl generated java files reside.

An important thing to note is: the tool generated many ObjectFactory.java files for serializer and deserializer to consume. You can find the location of those OjectFactory.java by the following command:

find . -name ObjectFactory.java

the output could be something like:

./com/yourinventit/processing/android/serial/ObjectFactory.java
./com/yourinventit/processing/android/ObjectFactory.java
./org/helloexample/ObjectFactory.java

2. SAAJ client
The SOAP with Attachments API for Java or SAAJ provides a standard way to send XML documents over the Internet from the Java platform.

The maven saaj dependency is:

<dependency>
    <groupId>com.sun.xml.messaging.saaj</groupId>
    <artifactId>saaj-impl</artifactId>
    <version>1.5</version>
</dependency>

With saaj you can create soap calls like the following. In order to request, we need to create the Request message. Follow the same way we used to fill the request form in soapUI GUI, here we need to fill many fields in the request SOAPMessage object as well. We fill these fields with the help of wsdl generated classes.

SOAPConnectionFactory soapConnectionFactory = SOAPConnectionFactory.newInstance();
SOAPConnection connection = soapConnectionFactory.createConnection();
URL endpoint = new URL("http://xyzcode.xyz/quote");
MyPayload payload = business.getRequest();  //pass in from call param
SOAPMessage  message = funcCreateRequest(payload);  //use wsdl generated classes
SOAPMessage response = connection.call(message, endpoint);
connection.close();

3. jaxb
We said that we fill soap request fields with the help of wsdl generated classes. In fact, there are java library to help serialize and deserialize your soap request and response. The tool for that is jaxb.
You can include jaxb using maven.

<dependency>
    <groupId>javax.xml.bind</groupId>
    <artifactId>jaxb-api</artifactId>
    <version>2.2.12</version>
</dependency>
<dependency>
    <groupId>com.sun.xml.bind</groupId>
    <artifactId>jaxb-impl</artifactId>
    <version>2.2.11</version>
</dependency>

Once you have jaxb, you need to tell JAXBContext where to find the ObjectFactory.java files in order to serialize and deserialize soap objects generated by the wsdl file.
With the above example, your initialization will looks like:

JAXBContext jc = JAXBContext.newInstance("com.yourinventit.processing.android.serial:com.yourinventit.processing.android:org.helloexample:org.helloexample");

Once you have the JAXB initialized, you can convert between object and xml string.
For request

MyPayload payload = business.getRequest();  //pass in from call param
Marshaller m = jc.createMarshaller();
StringWriter w = new StringWriter();
xsw = XMLOutputFactory.newInstance().createXMLStreamWriter(w);
m.marshal(payload, xsw);  //consume objects specified in ObjectFactory.java
String requestString = w.getBuffer().toString();
SOAPMessage soapMessage = funcBuildSoapRequest(requestString);

For response
Node payload = soapMessage.getSOAPBody().getFirstChild();
String responseString  = funcConvertPayloadToString(payload);
Unmarshaller u = jc.createUnMarshaller();
xsr = XMLInputFactory.newInstance().createXMLStreamReader(new StringReader(responseString));
Object o = unmarshaller.unmarshal(xsr);  //consume objects specified in ObjectFactory

The "Object o" can be cast to one of the wsdl object, then you can pass the object around in your program.

How to solve git merge conflict with merge rebase

A git repository usually have a stable master branch for building the release artifacts, a development branch for collecting development checkins. The developers branch out bugfix and feature branches from develop branch.

At 1 pm, developer A branched out from develop a bugfix1 branch and developer B branched out feature1 branch from develop. At 2 pm, developer A merged bugfix1 branch back to develop. At 3 pm, developer B also try to merge feature1 branch back to develop. The merge failed, the reason is they both modified the same lines for the same file and git can not tell whose version to keep. This situation is called merge conflict.

merge conflict at node4

To merge feature1 to develop, we use the following commands:
git checkout develop;
then
git merge feature1;

If the merge didn't success, the first thing we need to do is to issue
git merge --abort
on the develop branch to abort the merge. When you use bitbucket gui to create a pull request, the bitbucket do the "git merge --abort" for you on the develop branch automatically.

Next, we need to fix the feature1 branch so that the conflict won't be there when develop merge feature1 next time.

To do that, we can use the following commands, as we went through in the previous post:
git checkout feature1;
git merge develop;

This approach will create 3 logs in the develop branch after the merge: bugfix1 commit, feature1 commit, conflict resolving commit.

In this post, we will go through another approach, which is using:

git checkout feature1;
git rebase develop;

Git rebase command is one of the most powerful commands in Git. It has the ability to rewrite your repository's commit history, by rearranging, modifying, and even deleting commits.

rebase from node2 to node3

Here the rebase means to give a new parent to the feature1 branch. This is what you do when you use git rebase to keep up with the upstream repository. The most important reason for using git rebase here is to change the starting point of your local branch feature1. It previously has the starting point 2, after rebase, it's starting point becomes 3. Since the starting point changes, all the content from node3 is applied to your local copy and the conflict will be introduced. Then you manually modify the files to solve the conflict, then you got a new version on top of the node3 version to commit.

This approach will create 2 logs after develop merge the feature1 branch: bugfix1 commit, feature1 commit.  Notice there is no conflict solve commit.

Now you have the local changes ready for feature1 branch. You have 2 options to merge the changes back to develop branch.

If you use pure command line for merge, you can merge develop local, then push to remote repo.
You can just
git checkout develop;
git pull;
git merge feature1;
git push origin develop;

If you are using bitbucket and pull request for merge. You can push the feature1 changes to remote repo, then create pull request. That way, the git bucket pull request merge will execute the following command for you on the remote repository with bitbucket's credential instead of yours.
git checkout develop;
git merge feature1;


Since the rebase changed the starting point of feature1, the remote branch will get confused, so if you are using bitbucket, and using pull request to merge feature1 to develop, then you need to use the following command to push your local changes about feature1 to remote repo.

git push -f origin feature1;

The -f flag means you acknowledged the fact that the starting point of you local branch differ from the starting point of the remote branch for feature1.

The process is demonstrated in the following demo:



===============
GIT>git clone https://github.com/tekgadg/Pong_sept_20_2D.git demorepo
Cloning into 'demorepo'...
remote: Enumerating objects: 6, done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 6
Unpacking objects: 100% (6/6), done.
Checking connectivity... done.
GIT>cd demorepo/
GIT>echo a > a.txt
GIT>echo b > b.txt
GIT>echo c > c.txt
GIT>git branch develop
GIT>git checkout develop
Switched to branch 'develop'
GIT>git config --global --edit
GIT>git status
On branch develop
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add --all
GIT>git commit -m "develop base commit"
[develop e45abe7] develop base commit
 3 files changed, 3 insertions(+)
 create mode 100644 a.txt
 create mode 100644 b.txt
 create mode 100644 c.txt
GIT>git branch feature1
GIT>git checkout -b bugfix1 develop
Switched to a new branch 'bugfix1'
GIT>echo xyzcode > a.txt
GIT>echo xyzcode >> b.txt
GIT>echo xyzcode >> c.txt
GIT>git commit -am "bugfix1 commit"
[bugfix1 8488ffa] bugfix1 commit
 3 files changed, 3 insertions(+), 1 deletion(-)
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge bugfix1
Updating e45abe7..8488ffa
Fast-forward
 a.txt | 2 +-
 b.txt | 1 +
 c.txt | 1 +
 3 files changed, 3 insertions(+), 1 deletion(-)
GIT>git checkout feature1
Switched to branch 'feature1'
GIT>echo xyznetwork >> a.txt
GIT>echo xyznetwork > b.txt
GIT>git rm c.txt
rm 'c.txt'
GIT>git status
On branch feature1
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

deleted:    c.txt

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

modified:   a.txt
modified:   b.txt

GIT>git commit -am "feature1 commit"
[feature1 ce6291a] feature1 commit
 3 files changed, 2 insertions(+), 2 deletions(-)
 delete mode 100644 c.txt
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge feature1
CONFLICT (modify/delete): c.txt deleted in feature1 and modified in HEAD. Version HEAD of c.txt left in tree.
Auto-merging b.txt
CONFLICT (content): Merge conflict in b.txt
Auto-merging a.txt
CONFLICT (content): Merge conflict in a.txt
Automatic merge failed; fix conflicts and then commit the result.
GIT>git diff
diff --cc a.txt
index ae5111a,56bdbb4..0000000
--- a/a.txt
+++ b/a.txt
@@@ -1,1 -1,2 +1,6 @@@
++<<<<<<< HEAD
 +xyzcode
++=======
+ a
+ xyznetwork
++>>>>>>> feature1
diff --cc b.txt
index 5e2bb23,c09e41c..0000000
--- a/b.txt
+++ b/b.txt
@@@ -1,2 -1,1 +1,6 @@@
++<<<<<<< HEAD
 +b
 +xyzcode
++=======
+ xyznetwork
++>>>>>>> feature1
* Unmerged path c.txt
GIT>git merge --abort
GIT>git checkout feature1
Switched to branch 'feature1'
GIT>git rebase develop
First, rewinding head to replay your work on top of it...
Applying: feature1 commit
Using index info to reconstruct a base tree...
M a.txt
M b.txt
M c.txt
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): c.txt deleted in feature1 commit and modified in HEAD. Version HEAD of c.txt left in tree.
Auto-merging b.txt
CONFLICT (content): Merge conflict in b.txt
Auto-merging a.txt
CONFLICT (content): Merge conflict in a.txt
Failed to merge in the changes.
Patch failed at 0001 feature1 commit
The copy of the patch that failed is found in:
   /Users/homenetwork/git/demorepo/.git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".

a
GIT>git diff
xyznetwork
diff --cc a.txt
index ae5111a,56bdbb4..0000000
--- a/a.txt
+++ b/a.txt
@@@ -1,1 -1,2 +1,6 @@@
++<<<<<<< HEAD
 +xyzcode
++=======
+ a
+ xyznetwork
++>>>>>>> feature1 commit
diff --cc b.txt
index 5e2bb23,c09e41c..0000000
--- a/b.txt
+++ b/b.txt
@@@ -1,2 -1,1 +1,6 @@@
++<<<<<<< HEAD
 +b
 +xyzcode
++=======
+ xyznetwork
++>>>>>>> feature1 commit
* Unmerged path c.txt
GIT>vi a.txt
GIT>vi b.txt
GIT>git diff
diff --cc a.txt
index ae5111a,56bdbb4..0000000
--- a/a.txt
+++ b/a.txt
diff --cc b.txt
index 5e2bb23,c09e41c..0000000
--- a/b.txt
+++ b/b.txt
* Unmerged path c.txt
GIT>git status
rebase in progress; onto 8488ffa
You are currently rebasing branch 'feature1' on '8488ffa'.
  (fix conflicts and then run "git rebase --continue")
  (use "git rebase --skip" to skip this patch)
  (use "git rebase --abort" to check out the original branch)

Unmerged paths:
  (use "git reset HEAD <file>..." to unstage)
  (use "git add/rm <file>..." as appropriate to mark resolution)

both modified:   a.txt
both modified:   b.txt
deleted by them: c.txt

no changes added to commit (use "git add" and/or "git commit -a")
GIT>git log
commit 8488ffa7e153fe97fb6566f94f544a91e11b926b
Author: you <you@example.com>
Date:   Thu Feb 7 06:57:35 2019 -0500

    bugfix1 commit

commit e45abe719c91e6932329eaca3885f7f75715ac96
Author: you <you@example.com>
Date:   Thu Feb 7 06:54:31 2019 -0500

    develop base commit

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git add --all
GIT>git status
rebase in progress; onto 8488ffa
You are currently rebasing branch 'feature1' on '8488ffa'.
  (all conflicts fixed: run "git rebase --continue")

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

modified:   a.txt
modified:   b.txt

GIT>ls
AndroidManifest.xml a.txt c.txt sketch.properties
Pong_sept_20_2D.pde b.txt code
GIT>git add c.txt
GIT>git status
rebase in progress; onto 8488ffa
You are currently rebasing branch 'feature1' on '8488ffa'.
  (all conflicts fixed: run "git rebase --continue")

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

modified:   a.txt
modified:   b.txt

GIT>git rebase --continue
Applying: feature1 commit
GIT>git status
On branch feature1
nothing to commit, working directory clean
GIT>ls
AndroidManifest.xml a.txt c.txt sketch.properties
Pong_sept_20_2D.pde b.txt code
GIT>git log
commit 995868efe3b50a562b4015ec9aeb89fda585de1d
Author: you <you@example.com>
Date:   Thu Feb 7 06:59:33 2019 -0500

    feature1 commit

commit 8488ffa7e153fe97fb6566f94f544a91e11b926b
Author: you <you@example.com>
Date:   Thu Feb 7 06:57:35 2019 -0500

    bugfix1 commit

commit e45abe719c91e6932329eaca3885f7f75715ac96
Author: you <you@example.com>
Date:   Thu Feb 7 06:54:31 2019 -0500

    develop base commit

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge feature1
Updating 8488ffa..995868e
Fast-forward
 a.txt | 3 ++-
 b.txt | 3 +--
 2 files changed, 3 insertions(+), 3 deletions(-)
GIT>ls
AndroidManifest.xml a.txt c.txt sketch.properties
Pong_sept_20_2D.pde b.txt code
GIT>git log
commit 995868efe3b50a562b4015ec9aeb89fda585de1d
Author: you <you@example.com>
Date:   Thu Feb 7 06:59:33 2019 -0500

    feature1 commit

commit 8488ffa7e153fe97fb6566f94f544a91e11b926b
Author: you <you@example.com>
Date:   Thu Feb 7 06:57:35 2019 -0500

    bugfix1 commit

commit e45abe719c91e6932329eaca3885f7f75715ac96
Author: you <you@example.com>
Date:   Thu Feb 7 06:54:31 2019 -0500

    develop base commit

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git status
On branch develop
nothing to commit, working directory clean
GIT>

How to solve merge conflict with git merge

A git repository usually have a stable master branch for building the release artifacts, a development branch for collecting development checkins. The developers branch out bugfix and feature branches from develop branch.

At 1 pm, developer A branched out from develop a bugfix1 branch and developer B branched out feature1 branch from develop. At 2 pm, developer A merged bugfix1 branch back to develop. At 3 pm, developer B also try to merge feature1 branch back to develop. The merge failed, the reason is they both modified the same lines for the same file and git can not tell whose version to keep. This situation is called merge conflict.

merge conflict at node4

To merge feature1 to develop, we use the following commands:
git checkout develop;
then
git merge feature1;

If the merge didn't success, the first thing we need to do is to issue
git merge --abort
on the develop branch to abort the merge. When you use bitbucket gui to create a pull request, the bitbucket do the "git merge --abort" for you on the develop branch automatically.

Next, we need to fix the feature1 branch so that the conflict won't be there when develop merge feature1 next time.

To do that, we use the following commands:
git checkout feature1;
git merge develop;

This time instead of merge feature into develop, we merge develop into feature instead. As you can tell, no matter which direction the merge goes, the conflict will be there. We need to manually modify the conflicted files then commit a new version into feature1 branch. During our merge, the git knows our feature1 version is later than the develop version. So when we checkout develop then merge feature1, the feature1 version will override the develop version, so the merge will success.

Finally you use to sync your local and remote repo for the develop branch.
git push origin develop;

This approach will create 3 logs in the develop branch after the merge: bugfix1 commit, feature1 commit, conflict resolving commit.

For the developers that use bitbucket and pull request to merge feature and bugfix branches to develop. They need to solved the conflict local for feature1 branch, then
git push origin feature1;

then use pull request in bitbucket for merging into develop branch.

The process is demonstrated in the following demo:

In the next post, we will introduce an alternative approach with git rebase.
=============

GIT>git clone https://github.com/tekgadg/Pong_sept_20_2D.git demorepo
Cloning into 'demorepo'...
remote: Enumerating objects: 6, done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 6
Unpacking objects: 100% (6/6), done.
Checking connectivity... done.
GIT>cd demorepo/
GIT>git config --global user.email you@example.com
GIT>git config --global user.name "you"
GIT>git branch develop
GIT>git checkout develop
Switched to branch 'develop'
GIT>echo a > a.txt
GIT>ech b > b.txt
-bash: ech: command not found
GIT>echo b > b.txt
GIT>echo c > c.txt
GIT>git commit -am "develop base"
On branch develop
Untracked files:
a.txt
b.txt
c.txt

nothing added to commit but untracked files present
GIT>git status
On branch develop
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add .
GIT>git commit -m "develop base"
[develop 1a27b56] develop base
 3 files changed, 3 insertions(+)
 create mode 100644 a.txt
 create mode 100644 b.txt
 create mode 100644 c.txt
GIT>git branch feature1
GIT>git branch bugfix1
GIT>git branch
  bugfix1
* develop
  feature1
  master
GIT>git checkout bugfix1
Switched to branch 'bugfix1'
GIT>echo xyzcode >> a.txt
GIT>echo xyzcode >> b.txt
GIT>git rm c.txt
rm 'c.txt'
GIT>git diff
diff --git a/a.txt b/a.txt
index 7898192..8267385 100644
--- a/a.txt
+++ b/a.txt
@@ -1 +1,2 @@
 a
+xyzcode
diff --git a/b.txt b/b.txt
index 6178079..5e2bb23 100644
--- a/b.txt
+++ b/b.txt
@@ -1 +1,2 @@
 b
+xyzcode
GIT>git commit -am "bugfix1 commit"
[bugfix1 570873c] bugfix1 commit
 3 files changed, 2 insertions(+), 1 deletion(-)
 delete mode 100644 c.txt
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge bugfix1
Updating 1a27b56..570873c
Fast-forward
 a.txt | 1 +
 b.txt | 1 +
 c.txt | 1 -
 3 files changed, 2 insertions(+), 1 deletion(-)
 delete mode 100644 c.txt
GIT>git status
On branch develop
nothing to commit, working directory clean
GIT>git checkout feature1
Switched to branch 'feature1'
GIT>echo xyznetwork >> a.txt
GIT>echo xyznetwork > b.txt
GIT>echo xyznetwork >> c.txt
GIT>git commit -am "feature1 commit"
[feature1 6efd19f] feature1 commit
 3 files changed, 3 insertions(+), 1 deletion(-)
GIT>git diff
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge feature1
CONFLICT (modify/delete): c.txt deleted in HEAD and modified in feature1. Version feature1 of c.txt left in tree.
Auto-merging b.txt
CONFLICT (content): Merge conflict in b.txt
Auto-merging a.txt
CONFLICT (content): Merge conflict in a.txt
Automatic merge failed; fix conflicts and then commit the result.
GIT>git diff
diff --cc a.txt
index 8267385,56bdbb4..0000000
--- a/a.txt
+++ b/a.txt
@@@ -1,2 -1,2 +1,6 @@@
  a
++<<<<<<< HEAD
 +xyzcode
++=======
+ xyznetwork
++>>>>>>> feature1
diff --cc b.txt
index 5e2bb23,c09e41c..0000000
--- a/b.txt
+++ b/b.txt
@@@ -1,2 -1,1 +1,6 @@@
++<<<<<<< HEAD
 +b
 +xyzcode
++=======
+ xyznetwork
++>>>>>>> feature1
* Unmerged path c.txt
GIT>git merge --abort
GIT>git checkout feature1
Switched to branch 'feature1'
GIT>git checkout develop
Switched to branch 'develop'
GIT>git diff
GIT>git checkout feature1
Switched to branch 'feature1'
GIT>git merge develop
CONFLICT (modify/delete): c.txt deleted in develop and modified in HEAD. Version HEAD of c.txt left in tree.
Auto-merging b.txt
CONFLICT (content): Merge conflict in b.txt
Auto-merging a.txt
CONFLICT (content): Merge conflict in a.txt
Automatic merge failed; fix conflicts and then commit the result.
a
GIT>git diff
xyznetwork
diff --cc a.txt
index 56bdbb4,8267385..0000000
--- a/a.txt
+++ b/a.txt
@@@ -1,2 -1,2 +1,6 @@@
  a
++<<<<<<< HEAD
 +xyznetwork
++=======
+ xyzcode
++>>>>>>> develop
diff --cc b.txt
index c09e41c,5e2bb23..0000000
--- a/b.txt
+++ b/b.txt
@@@ -1,1 -1,2 +1,6 @@@
++<<<<<<< HEAD
 +xyznetwork
++=======
+ b
+ xyzcode
++>>>>>>> develop
* Unmerged path c.txt
GIT>vi a.txt
GIT>vi b.txt
GIT>git rm c.txt
c.txt: needs merge
rm 'c.txt'
GIT>git diff
diff --cc a.txt
index 56bdbb4,8267385..0000000
--- a/a.txt
+++ b/a.txt
diff --cc b.txt
index c09e41c,5e2bb23..0000000
--- a/b.txt
+++ b/b.txt
@@@ -1,1 -1,2 +1,3 @@@
 +xyznetwork
+ b
+ xyzcode
GIT>git status
On branch feature1
You have unmerged paths.
  (fix conflicts and run "git commit")

Changes to be committed:

deleted:    c.txt

Unmerged paths:
  (use "git add <file>..." to mark resolution)

both modified:   a.txt
both modified:   b.txt

GIT>git commit -am "conflict resolve with merge"
[feature1 b863a0d] conflict resolve with merge
GIT>git checkout develop
Switched to branch 'develop'
GIT>git merge feature1
Updating 570873c..b863a0d
Fast-forward
 b.txt | 1 +
 1 file changed, 1 insertion(+)
GIT>git status
On branch develop
nothing to commit, working directory clean
GIT>git branch
  bugfix1
* develop
  feature1
  master
GIT>git branch --delete bugfix1 feature1
Deleted branch bugfix1 (was 570873c).
Deleted branch feature1 (was b863a0d).
GIT>git branch
* develop
  master
GIT>git log
commit b863a0d537b6201495e4a7169883f32180b07f0e
Merge: 6efd19f 570873c
Author: you <you@example.com>
Date:   Wed Feb 6 07:44:21 2019 -0500

    conflict resolve with merge

commit 6efd19f7745211562b3ad33d079c55aae6beeb43
Author: you <you@example.com>
Date:   Wed Feb 6 07:40:53 2019 -0500

    feature1 commit

commit 570873c48d0e69c6a45e9b7d5891c7c35937c300
Author: you <you@example.com>
Date:   Wed Feb 6 07:39:04 2019 -0500

    bugfix1 commit

commit 1a27b5688d18352f747e7eea7ada7c65aeb8e90c
Author: you <you@example.com>
Date:   Wed Feb 6 07:37:08 2019 -0500

    develop base

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>

7 tools that monitor your servers and web applications

Your web application is a distributed software system that runs 24/7 on a fleets of servers. There are many functional components such as DNS server, load balancer, router, web server, authentication service, provisioning service, redex server, database server etc. Constantly they are generating data such as request count, response time, cpu load, ram usage, disk usage, web container thread pool size, application thread pool size, database connection pool size, garbage collection time, etc. For each data source, you can get static data such as average, sum, min, max, perc99, perc95, etc. You can then sort them by site, hostname, IP, date, tag, source etc. Sometimes, there are correlations between data types. For example high request count often correlates to high cpu load, high ram, high thread-pool size and longer database response time, etc.

The following 7 tools will help you to collect the data from your application and organize them in smart ways, so that your machines are under control.

1. Event queues

Your web application is a group of applications. Each application is deployed on a fleet of servers behind load balancer, and each copy of the artifact deployed on each server is a multi-threading program. As you can tell, there are high chance for race conditions. We don't want our application meta data collection ruin the main functionality or create synchronize bottleneck. A thread-safe queue implementation can help a lot in this situation. It decouples producer and consumer, so that the log generated from one thread in one VM won't mess up with another thread in the same VM or another VM. For example Kafka and RabbitMQ are two of this kind.  Your application can send events as object to a topic/stream of the queue. These topics, after processing, can be sent to the API of consumers such as another app, database, zabbix, splunk, tableau etc.

2. Data analytic tools

Your Kafka stream can be sent to data analytic tools such as splunk. Once the data get there, they are stored in the way fast retrieving and sorting are possible. Splunk can analyze huge amount of data, use them to create statistics, time charts and dashboards. Splunk also allows events to trigger alerts. The alert threshold criteria can range from count larger than a number to time longer than a value. Once the alert is triggered, actions can be taken, for example, send message to slack channel, email, SMS, ticketing API.

3. Incident Response Platform

There are many ticketing platforms such as bmc remedy, salesforce, pagerduty. Pagerduty for example, allows you to install an app in splunk. One of the splunk alert action is send to a pagerduty service. Once the pagerduty service receives a call, it acts according to the escalation policy -- for example, call the primary contacts, if not get acknowledgement within 5 minutes, call the secondary contacts, so on and so forth, until someone takes action about the event-- could be annoying during night hours.

4. Host inventory tools

Your server warehouse need management. Hardware as well as VMs can be managed with tools such as zabbix. Once the zabbix agent is installed on an unix server, it starts to collect infrastructure informations such as cpu, memory, hdc io, hdc bw, sda io, sda bw, etc. It allows monitor, inventory and report about infrastructure hosts on the Zabbix server. You can also draw graph with your hardware resource usage history.

5. VM and cloud inventory tools

Virtual machines and cloud VM instances are special resources. They are special because they are elastic and volatile. Tools such as VMWare Vcenter/Vsphere, AWS, GCP, AZURE allows you to quickly create/destroy VMs, create cloud virtual company, reconfigure routing rule, setup ACL, configure scale up/down policy etc. Oracle has its own JVM debug tool which can be invoked with command JVisualVM.

6. Data warehouse mining and visualization tools

Tools such as tableau can connect to almost any data warehouse applications on the market: oracle, mysql, AWS Redshift, cubes, Teradata, cassandra, data lake, redis, microsoft SQL Server, mongodb, hyperion, etc. With the wealth of data already stored in the data warehouse, tableau can generate reports, statistics, graphs across different data sources and give the user power to further analyze them.

7. Network inventory tool

Your network resources such as IPs, Nodes, Servers need to be organized and grouped. There are tools such as NGINX, BIG IP to help you out. At anytime, you are able to inventory your network resource by locations, OS, liveness, cells etc. You are also able to take a group of servers out of service or put them into service.

7 git undo commands you should know

In git, you can create new files then add them into git staging area, then commit them into local repo history. At any time, you can undo those git adds and commits.

There are a group of commands that allow you undo:

1. rm -rf {filename}
2. git reset HEAD
3. git reset HEAD {filename}
4. git reset --hard HEAD
5. git reset --hard origin/{branchname}
6. git checkout -f
7. git revert HEAD

First of all untracked files should not be the git's concern, unless you use rm {filename} to remove them from your disk, any commands start with git (except git clean) should not touch them.

git reset HEAD
this command put the files in the staging area back to the untracked state.

git checkout -f
this command will replace the local files with the remote version. Of course, your untracked files will remain untouched. It is more destructive than git reset HEAD, since the files in staging area are wiped out from disk, they are not put back into untracked state.

notice Neither "git checkout -f", "git reset HEAD" nor "git reset --hard HEAD" will touch the committed files.
In order to undo commit, we need to issue command "git reset --hard orign/{branchname}". This command is destructive -- it won't put the committed files back to untracked the state, it wipes them away from the disk.

Another way to undo commit is to use
git revert HEAD
it added a new commit to undo the local commits so that the end result is the same of HEAD version. It is a safer way of undo commit since it kept a copy of you committed files in history.

exmple

============

GIT>git clone https://github.com/tekgadg/Pong_sept_20_2D.git demorepo
Cloning into 'demorepo'...
remote: Enumerating objects: 6, done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 6
Unpacking objects: 100% (6/6), done.
Checking connectivity... done.
GIT>cd demorepo/
GIT>ls
AndroidManifest.xml code
Pong_sept_20_2D.pde sketch.properties
GIT>echo a > a.txt
GIT>echo b > b.txt
GIT>echo c > c.txt
GIT>mkdir test
GIT>echo d > test/d.txt
GIT>echo e > test/e.txt
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add --all
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   a.txt
new file:   b.txt
new file:   c.txt
new file:   test/d.txt
new file:   test/e.txt

GIT>git reset HEAD test/
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   a.txt
new file:   b.txt
new file:   c.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

test/

GIT>git checkout -f
Your branch is up-to-date with 'origin/master'.
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>ls
AndroidManifest.xml code test
Pong_sept_20_2D.pde sketch.properties
GIT>echo a > a.txt
GIT>echo b > b.txt
GIT>echo c > c.txt
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add -u .
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add .
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   a.txt
new file:   b.txt
new file:   c.txt
new file:   test/d.txt
new file:   test/e.txt

GIT>git reset HEAD c.txt test
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   a.txt
new file:   b.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

c.txt
test/

GIT>git commit -m "step 1 commit 2 files"
[master 6ff2dba] step 1 commit 2 files
 Committer: Home Network <homenetwork@OK-MBP.home>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:

    git config --global --edit

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 2 files changed, 2 insertions(+)
 create mode 100644 a.txt
 create mode 100644 b.txt
GIT>git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
Untracked files:
  (use "git add <file>..." to include in what will be committed)

c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git log
commit 6ff2dba507fd244988282078e349f34807ca9bb9
Author: Home Network <homenetwork@OK-MBP.home>
Date:   Tue Feb 5 08:14:49 2019 -0500

    step 1 commit 2 files

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git reset HEAD
GIT>git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
Untracked files:
  (use "git add <file>..." to include in what will be committed)

c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git log
commit 6ff2dba507fd244988282078e349f34807ca9bb9
Author: Home Network <homenetwork@OK-MBP.home>
Date:   Tue Feb 5 08:14:49 2019 -0500

    step 1 commit 2 files

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git reset --hard HEAD
HEAD is now at 6ff2dba step 1 commit 2 files
GIT>git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
Untracked files:
  (use "git add <file>..." to include in what will be committed)

c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git log
commit 6ff2dba507fd244988282078e349f34807ca9bb9
Author: Home Network <homenetwork@OK-MBP.home>
Date:   Tue Feb 5 08:14:49 2019 -0500

    step 1 commit 2 files

commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>git reset --hard origin/master
HEAD is now at 40aac7d upload
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git log
commit 40aac7d161dc232299f81ed7112b88986b5fbe39
Author: SeanShin <seanshin@RetOne.local>
Date:   Sun Jun 14 16:19:04 2015 -0700

    upload
GIT>ls
AndroidManifest.xml c.txt sketch.properties
Pong_sept_20_2D.pde code test
GIT>echo a > a.txt
GIT>echo b > b.txt
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt
test/

nothing added to commit but untracked files present (use "git add" to track)
GIT>git add test/
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   test/d.txt
new file:   test/e.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt

GIT>git checkout -f
Your branch is up-to-date with 'origin/master'.
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt
c.txt

nothing added to commit but untracked files present (use "git add" to track)
GIT>ls
AndroidManifest.xml b.txt sketch.properties
Pong_sept_20_2D.pde c.txt
a.txt code
GIT>git add c.txt
GIT>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

new file:   c.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

a.txt
b.txt