How to cat multiline content into a file with shell

Sometimes we want to copy a paragraph then paste into or append into a file.

We can use a text editor, such as vi to do the copy paste.

It will be convenient to do it in command line with just cat and >.

Here is the example, the normal way is echo some thing followed by >. 

In case of multi-line, we need to use <<EOF to mark the end of the file.

EOF won't be a literal string that enters the final result, it is the marker of End Of File instead.

here is the example code:

demo>cat > demo.txt <<EOF

> adfd

> dfadfad

> adfafa

> EOF

demo>cat demo.txt

adfd

dfadfad

adfafa

demo>

The file content is now get updated with your strings.

How to build rpms

prepare the source file

first of all, you need all your source files ready under ~/RPMBUILD/SOURCES dircetory
For example if you wan to build jkd1.8.0 rpm, you may want the java8-local_policy.jar and java8-US_export_policy.jar from java 8 JCE and the jre-8u111-linux-x64.tar.gz from oracle website.

prepare the .spec file

once the source files are in place, you need to write a mybuild.spec file to instruct the rpm command how to pre, build, install, clean, files, preun, post the files in the rpm package.
this file has to be put in the ~/RPMBUILD/SPECS directory.

build rpm

now you can run the following command to build the rpm

rpmbuild -ba ~/RPMBUILD/SPECS/mybuild.spec

this command will put the resulting rpm in ~/RPMBUILD/RPMS/x86_64.

post to yum repo

You can optional set the rpm to be managed by yum.

Assuming you plan to place the repository in /var/www/html which is the default directory for Red Hat based Apache installations.

Also assuming you have run 

yum install yum-utils;

reposync -p /var/www/html/;

so that you can present this directory with apache.

now all you have to do is copy the rpm into /var/www/html/fedora/jdk8/

then run command

run createrepo /var/www/html/fedora/jdk8/

If your apache server is running, your new rpm should now be managed by yum.

If not, run it:

yum install httpd;

chekconfig httpd on;

service httpd start;

config yum client

On the client host you need to point the yum repo to the yum server you just setup,

for simplicity, lets use the yum server itself as client

create a /etc/yum.repos.d/local.repo file

[Local-Repository]

name=mylocal

baseurl=http://ip-of-my-server/fedora/jdk8

enabled=1

gpgcheck=0

final result

now once you type 

yum install rpm-name-set-in-spec-file

the rpm will be intalled

How to generate date string with shell command

Sometimes we need to generate date string, the shell command default format can brings us somewhere, but how about the date with format 20-05-10?
We can print date string using shell command date with format flags.

date +%y-%m-%d
for example will print today's date in short format as
20-05-13

sometimes, the generate date strings can be used to construct other shell commands with ``.
The date command wrapping inside `` will get executed first, the result string then concatenated with the rest of the strings to form a shell command.

For example, when the following command is executed

echo filename.`date +%y-%m-%d`.log

`date +%y-%m-%d` will execute first, generate a string 20-05-13. Then string 20-05-13 is concatenated with rest of the command to get a new shell command:

echo filename.20-05-13.log

the above command's running result is
filename.20-05-13.log

 

=================
demo>date
Wed May 13 15:05:10 EDT 2020
demo>date +%y-%m-%d
20-05-13
demo>echo filename.`date +%y-%m-%d`.log
filename.20-05-13.log
==================

Similarly, we can use the same technique to construct more sophisticated grep shell command:

ls myapp.`date +%y-%m-%d`*.log | xargs -I {} cat{} | grep -i 'exception\|error' -A 2 -B 2

the above command, list all myapp log files generated today. Among though files, we search for exception and error messages, print out the line with 2 lines before and 2 lines after.

How to create a new file in shell script with cat EOF

demo>cat test.sh
#!/bin/bash
rm -f temp_config

cat > temp_config <<-EOF
sfdfadfadfadf
dfadfadfad
[xxxx]
xfdfdf
EOF

cat temp_config
ls -lrt temp_config
demo>
demo>./test.sh
sfdfadfadfadf
dfadfadfad
[xxxx]
xfdfdf
-rw-r--r--  1 homenetwork  staff  39 May 12 14:52 temp_config
demo>

7 shell commands for network trouble-shooting

1. nslookup <url>
check what's your url is known by the dns server.

2. ping <IP>
Is the IP alive? If ping get 100% loss, it could mean two things, either the host is down or it is blocking the ICMP traffic.

3. curl <url>
check if the web application you are trying to access is alive.

4. netstat -nultp
ssh into the server and check what's going on there. run the command on the server machine to list all the listening ports of your application, for example, your java program's listening ports are:

netstat -nultp | grep java

5. nc <IP> <port>
Your web application might be listening on a port such as 8799, which you don't know how to communicate with. Use netcat, the general purpose net client to connect to the port to tell if the port is open.

6. openssl s_client -connect <IP:port>
Whenever your web application get authentication issue during network connection, get the public key from the destination server, then check whatever your web client was looking for are there.

7. tcpdump -i eth0 -s0 -v port 80
Still have issue? Try to get the pcap from the source host, load balancer, proxy server, destination host, then look into them, looking for issues such as connection reset, socket reuse, traffic drop, wait timed out, concurrency bug, etc.

7 techniques that help system administrator to find bottleneck

Your web application runs slow and your clients are complaining about long response time.

You need to figure out what's the bottleneck of your program.

• is it cpu bound?
• is it memory bound?
• is it disk space bound?
• is it network io bound?

Here are 7 techniques to help you find out.

1. shell command "uptime"

demo>uptime

18:30  up 24 days, 12:18, 3 users, load averages: 1.66 2.31 7.74

in the output, the most important information is the load averages: it is the 1 minute, 5 minute and 15 minute load average respectively. This number need to be make sense together with your cpu number.

You can find your cpu number with command:

grep -c '^processor' /proc/cpuinfo

For example, you 15 minutes load average is 7.74 and you have 2 cores. Then each core have roughly 4 processes running on it. This is far overloaded. However, if you have 8 cores, each process get one core, your cpu is running at the sweet spot. If you have 16 cores, there are about 8 cores idle, you didn't utilize your cpu resources cost-effectively.

the load average can also be found from command

top

2. shell command "free -m" 

free -m total used free shared buff/cache availableMem: 1695 517 249 0 927 1010Swap: 0 0 0

the output gives you an estimation of the memory usage in M. In the above example, you have 1010M RAM available, which is 2 times more than the amount used. Your server is not memory bound.

Again, this information can be found from the command

top

3. shell command "df -h"

df -hFilesystem Size Used Avail Use% Mounted onoverlay 36G 25G 11G 71% /tmpfs 64M 0 64M 0% /devtmpfs 848M 0 848M 0% /sys/fs/cgroup/dev/sdb1 4.8G 11M 4.6G 1% /home/dev/sda1 36G 25G 11G 71% /rootoverlayfs 1.0M 124K 900K 13% /etc/ssh/keystmpfs 848M 736K 847M 1% /run/metricsshm 64M 0 64M 0% /dev/shmoverlayfs 1.0M 124K 900K 13% /etc/ssh/ssh_host_rsa_keytmpfs 848M 0 848M 0% /run/google/devshell

The output of df let you know your disk space usage. 

In the above example, the first row tells you this: you have a 36G storage from a file system named "overlay", it is mounted on dirctory /. 71% of the disk space on that file system has been used. If you want to clean up some space there, you can 

cd /

then find the largest files occupying the space then decide which ones you want to rm.

du -h . | sort -nr | head

Similarly, the 5th row tells you: you have another 36G storage from a file system named /dev/sda1, it is mounted on directory /root. 71% of the space on that file system has been used. If you want to clean up some space there, you can 

cd /root

then find the largest files occupying the space then decide which ones you want to rm.

du -h . | sort -nr | head

4. shell command "sar -u" and "iostat -x 3"

these commands will gives you historical/realtime information about cpu/memory/disk usage, which further helps you look into hardware resource related bottleneck.

For example, if your web application is running in a VM, that VM might be hosted in a hypervisor. You have to consider the hypervisor when you analyze performance. Your hypervisor could be overloaded if the administrator put tons of VMs on one machine, or that guy simply put the hypervisor in power save mode to save bucks. You can find the tell-tell sign such as abnormal %iowait in "sar -u" output. Once you found something smelly, you can have the VM administrators to check the VM stats such as READY and Co-Stop for signs of overloading.

5. shell command "ps -ef | grep java"

the output could gives you the resource usage of your java web application, or even better, the JVM parameters of your java program. Look for parameter such as

• -server         optimized for server application
• -Xms           minimum heap size
• -Xmx           maximum heap size
• -XX:NewRatio                     ratio of young and old generation section
• -XX:+UseG1GC                  garbage collection strategy
• -XX:-DisableExplicitGC     prevent stop of the world full garbage collection

6. check the web container resource usage

For example, you might have tomcat as your web container. Your web container's thread pool size might be too large or too small. The web container could either wasted too much memory with a over-sized thread pool, or have the requests queued up with a too small thread pool.

7. check the web application resource usage

Have your application log the performance matrix such gc stats, cpu stats, thread pool size, database connection pool size, queue size, etc. then seek optimization opportunities. 

For example:

• thread pool size should be decided according to the requests per second and the response time. For example, your host gets 10 requests per second, each requests need average 2 seconds to serve. You should have at least 20 threads if you don't want arriving requests to pile up waiting for service. It is a rule of thumb, 2 threads will be too few, 200 threads will be too many. You benchmark and test in order to find the number between 2 and 200.
• if your gc happened too often or took too long, you need a bigger heap size or change the garbage collecting strategy.
• if your threads spend most of time waiting for IO, increasing thread pool size or increasing heap size won't help. 
• if your database query took too long, you either need a bigger database pool or explore more parallelization to fetch from database earlier, or you need to cache some contents local.
• if your threads spends most of time idle, you want to shrink the thread pool size or increase supply. For example, giving a bigger job queue, so that there are always job available to fetch even though the requests arriving rates fluctuate. Notice an over-sized job queue could boost performance on a single host but harm overall performance in a host farm. Smaller job queue encourage more fair traffic distribution among the hosts, while larger job queue could cause one host to be overloaded, by chance, when there are spikes in the traffic.
• if your threads spent time do task in serial, such as request another network service one by one, then find an opportunity for parallelization. Create a dedicated thread pool just to change that serial computation into a parallel one. 
• if two computation steps don't have to be coupled, decouple them with a concurrent queue, then have a producer thread pool to feed the queue then have a consumer thread pool to consume the queue.

7 tools that monitor your servers and web applications

Your web application is a distributed software system that runs 24/7 on a fleets of servers. There are many functional components such as DNS server, load balancer, router, web server, authentication service, provisioning service, redex server, database server etc. Constantly they are generating data such as request count, response time, cpu load, ram usage, disk usage, web container thread pool size, application thread pool size, database connection pool size, garbage collection time, etc. For each data source, you can get static data such as average, sum, min, max, perc99, perc95, etc. You can then sort them by site, hostname, IP, date, tag, source etc. Sometimes, there are correlations between data types. For example high request count often correlates to high cpu load, high ram, high thread-pool size and longer database response time, etc.

The following 7 tools will help you to collect the data from your application and organize them in smart ways, so that your machines are under control.

1. Event queues

Your web application is a group of applications. Each application is deployed on a fleet of servers behind load balancer, and each copy of the artifact deployed on each server is a multi-threading program. As you can tell, there are high chance for race conditions. We don't want our application meta data collection ruin the main functionality or create synchronize bottleneck. A thread-safe queue implementation can help a lot in this situation. It decouples producer and consumer, so that the log generated from one thread in one VM won't mess up with another thread in the same VM or another VM. For example Kafka and RabbitMQ are two of this kind.  Your application can send events as object to a topic/stream of the queue. These topics, after processing, can be sent to the API of consumers such as another app, database, zabbix, splunk, tableau etc.

2. Data analytic tools

Your Kafka stream can be sent to data analytic tools such as splunk. Once the data get there, they are stored in the way fast retrieving and sorting are possible. Splunk can analyze huge amount of data, use them to create statistics, time charts and dashboards. Splunk also allows events to trigger alerts. The alert threshold criteria can range from count larger than a number to time longer than a value. Once the alert is triggered, actions can be taken, for example, send message to slack channel, email, SMS, ticketing API.

3. Incident Response Platform

There are many ticketing platforms such as bmc remedy, salesforce, pagerduty. Pagerduty for example, allows you to install an app in splunk. One of the splunk alert action is send to a pagerduty service. Once the pagerduty service receives a call, it acts according to the escalation policy -- for example, call the primary contacts, if not get acknowledgement within 5 minutes, call the secondary contacts, so on and so forth, until someone takes action about the event-- could be annoying during night hours.

4. Host inventory tools

Your server warehouse need management. Hardware as well as VMs can be managed with tools such as zabbix. Once the zabbix agent is installed on an unix server, it starts to collect infrastructure informations such as cpu, memory, hdc io, hdc bw, sda io, sda bw, etc. It allows monitor, inventory and report about infrastructure hosts on the Zabbix server. You can also draw graph with your hardware resource usage history.

5. VM and cloud inventory tools

Virtual machines and cloud VM instances are special resources. They are special because they are elastic and volatile. Tools such as VMWare Vcenter/Vsphere, AWS, GCP, AZURE allows you to quickly create/destroy VMs, create cloud virtual company, reconfigure routing rule, setup ACL, configure scale up/down policy etc. Oracle has its own JVM debug tool which can be invoked with command JVisualVM.

6. Data warehouse mining and visualization tools

Tools such as tableau can connect to almost any data warehouse applications on the market: oracle, mysql, AWS Redshift, cubes, Teradata, cassandra, data lake, redis, microsoft SQL Server, mongodb, hyperion, etc. With the wealth of data already stored in the data warehouse, tableau can generate reports, statistics, graphs across different data sources and give the user power to further analyze them.

7. Network inventory tool

Your network resources such as IPs, Nodes, Servers need to be organized and grouped. There are tools such as NGINX, BIG IP to help you out. At anytime, you are able to inventory your network resource by locations, OS, liveness, cells etc. You are also able to take a group of servers out of service or put them into service.

maven trouble-shooting checklist

In theory, in order to use maven to build a project, all you need to do is to download maven, then update ~/.bashrc to have maven bin in $PATH.

export PATH=/Users/youruser/Downloads/apache-maven-3.2.5/bin/:$PATH

However, in reality, sometimes your maven gets stuck when downloading dependencies. Don't blame yourself for fat fingers, for big projects, the maven build process different every single time! First of all, your dependency could have snapshot versions, which might get updated on the repo 5 minutes ago, so when maven downloaded the latest one, trouble is introduced. Even though your direct dependency have no snapshot version. Your indirect dependency could have snapshot version dependency, downloading them also brings new jars which brings trouble into your build process. Besides, your OS/network admin could applied rules on your account, so that some of your remote repos can no longer be reached. Your build will still go on because your cached jar in local repo, however the cached version is now differ from the latest version, so your build process will differ from your coworkers' build process. Even though nothing happens in the environment, your download could be erroneous during file transfer due to network package loss, memory buffer, stale file stub etc.

Here are a trouble-shooting checklist.

1. find which maven is used.

which mvn

This command also tells you the maven home location.

There are two locations where a settings.xml file may live:
The Maven install: ${maven.home}/conf/settings.xml
A user’s install: ~/.m2/settings.xml
The user's install takes precedence than the other.

check both settings.xml file to make sure what properties are override. Without override, you default maven repo should locate inside ~/.m2/repository

2.  find if the complained jar file is in your local and remote repo or not.
The maven build will check local cache first, then go to download the remote repo for snapshot versions and won't download the release versions.
If the jar file is in the local repository ~/.m2/repository. It could be corrupted during download. You can use
mvn clean -U package
to force maven to re-download the existing local repository files from remote repository.
To achieve the same goal, you can also delete that dependency folder, or even the entire ~/.m2/repository folder, so that maven can not find the jar in local cache and have to trip to the remote repository to re-download everything.

Notice often times, the maven is complain about a jar and its depended jar file chains -- check each one of them, not just the first one in the complained list!

If you are helping others to trouble shoot build issue and the complained jar files are in your local repo but not in the remote repo. Chances are it just builds for you but won't build for others. Because someone has deleted the needed jar files in the remote repository, leaving a bug in the unaware victim code base.

3. restart you computer if possible. It will fix weird issues related to memory and network policy etc. For example network policy changes when you work from home with vpn then work in office.

4. check eclipse maven settings.

in eclipse -> preferences -> maven -> user settings, make sure you didn't override any default settings, so that you are using the default local repository. If you don't want to rely on default, literally specify the default location so that you are sure.

right click the project, then "maven", then "update project..", you can check the force update check box to have eclipse do the equivalent of
mvn clean -U package

5. restart eclipse, eclipse could run into weird issue internally.

6. ask if other people has the same build issue, then ask them to send you the ~/.bashrc, ~/.bash_profile, ~/.m2/settings.xml, and the /path-to-maven-home/conf/settings.xml. Also ask for the result of ls -lrt ~/.m2/repository/path/to/complained/jar

7. if you are using git or svn, find out who added the dependency that causes the trouble, have a conversation about it.

svn to git migration

Migrating svn to git needs some plan. There is a period, when the new git repo is created but the developers are continue to commit to the old svn repo.

Bitbucket provided a tool called svn-migration-scripts.jar to make the process easy. Down load the jar file and put under home directory ~/.

5 steps migration.

step 1. create a case-insensitive file-system image. 

If you are using linux such as redhat, ubuntu, your system already case-sensitive. If you are using mac os X, your system is using case-insensitive file-system. Run the following command to tell:

java -jar ~/svn-migration-scripts.jar verify

If the output looks like this, you then need to create a image.
"You appear to be running on a case-insensitive file-system. This is unsupported, and can result in data loss."

java -jar ~/svn-migration-scripts.jar create-disk-image 5 GitMigration

once the directory is created, cd there

cd ~/GitMigration

step 2. Extract author information

java -jar ~/svn-migration-scripts.jar authors https://svn.example.com > authors.txt

modify the emails in authors.txt

step 3. clone the svn to git repo

if your svn don't have a trunk, branch, tag lay out, use the following command.

git svn clone -authors-file=authors.txt https://svn.yoursvnrepo.com/YourSvnProj YourSvnProjAsGit

otherwise, you can specify the location of your trunk, branches, tag, use the following command instead

git svn clone --trunk=/trunk --branches=/branches --branches=/bugfixes --tags=/tags --authors-file=authors.txt https://svn.yoursvnrepo.com/YourSvnProj YourSvnProjAsGit

now clean the created project.

java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git --force

step 4. upload to git repo

Once you created a new git repo in bitbucket, you can push your local repo to the remote repo.

git remote add origin https://<user>@bitbucket.org/<user>/<repo>.git

git push -u origin --all

git push --tags

step 5. start to use git repo

now your coworker can start to clone the git repo and work on it.

git clone https://<user>@bitbucket.org/<user>/<project>.git <destination>

for the users that are still committing to the old svn, use the following command to synchronize

git config svn.authorsfile <path-to-authors-file>
git svn fetch
java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar sync-rebase
java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git --force

7 useful mac os x command line tools

Mac OS X has Darwin Unix like os. There are command line tools you can use in the console.

1. brew is a command to install other command line tools for mac os x.
To install brew and add the brew binary into the PATH, run the following commands in the console.

mkdir homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C homebrew;

echo export PATH=`pwd`/homebrew/bin:'$PATH' >> ~/.bashrc;

source ~/.bashrc;

which brew;

The last command "which brew;" should give the full path to brew binary file.

Now you have brew command available, you can install other command line tools easily.

2. pigz is a tool that is gzip equivalent but runs faster by exploiting parallel computing. pigz is useful when you need to unzip contents with gzip or zlib format. For example, you rabbitmq messages might be zlib compressed, then base64 encoded. Your decoding commands might looks funny:
base64 -D input.txt | pigz -d -z -c

To install pigz, issue the following command:
brew install pigz

3. apache bench (ab) is a command line tool that allows you to bench mark web sites. The most useful flag is -c, which allows you to specify the number of concurrent http requests sent to the host each time.
For example:

ab -n 100000 -c 10 -t 30 http://xyzcode.blogspot.com/

the above commands send total 10000 requests to xyzcode.blogspot.com, each time, 10 concurrent requests are sent.

ab is installed by default on mac os x, type which ab to check out. In case it is missing, you can install it with command:
brew install ab

4. pybot is a command line tool that automate stuff like ssh, web browsing etc. It is equivalent to have a robot to type the keyboard and read the screen then act upon it for you.

To install pybot, issue the following command:
brew install robot-framework

Since brew also installed python's package management tool, you can also use pip to install pybot:
pip install robotframework

5. in case pybot is an overkill for you, you can use expect command to automate ssh chat with remote hosts like a robot. Expect commands expect input, then send the response without any user interaction.
expect script is installed by default, check the installation path with
which expect;

6. zgrep is a better tool than grep.
zgrep allows you to grep content in zip files and normal files. It will be useful to search something in current log and backup logs.
The flexible syntax really compliments your searching strategy. So if you know the log will looks like

2018-11-07 16:42:25-05 HostNameppp Sophos Installer[72881]: [SGCCDFSBroker.m:306] Feedback json file was successfully uploaded (status code: 201).

Your searching strategy could be:
zgrep -i 'Sophos.*json.*successful' /var/log/install.log

each .* separated string became a filter string, so that you can narrow down the search.

You can use or syntax like:
zgrep -i 'Sophos.*json.*successful' /var/log/install.log | grep --color '72881\|72889'

zgrep is installed by default
which zgrep; will give you the installation location.

7. mysql is a command line tool to query mysql database. It allows you scripting mysql queries. You can pipe the result into zgrep to search relevant information.

mysql --host mysql.local -uuser -ppass -e 'select * from ct_prov.Student\G'

to install mysql command line tool
brew install mysql

tools for handling multiple ssh on mac os x

If you ever worked in cooperate environment or be the network administrator, you sometimes need to login many servers and check stuff simultaneously.

On windows, you can use putty or smart putty to handle multiple ssh connections.

Here are some tools for mac os x equivalent.

csshx

installation:

brew install csshx

usage:

csshx --login username host1 host2 host3 host4

xterm

installation:

download the mac version from the xterm website www.iterm2.com 

usage:

add a connection profiles:

click Profiles -> Open Profiles... -> Edit Profiles -> click the + sign to add a list of hostnames and ip addresses you would like to connect to -> when creating a host record, select command radio button then type in ssh command.

open multiple profiles simultaneously in tabs:

click Profiles -> Open Profiles... -> use shift or control key to hightlight a list of hosts you want to connect to -> click New Tab

The highlighted hosts will be connected and open in tabs.

send commands to multiple tabs:

click Shell -> Broadcast Input -> Broadcast Input to All Panels in All Tabs

7 eclipse debugging tips you should know

You may already know that you can set a breakpoint in eclipse for your java code, then have the program stop at the breakpoint in debug perspective, so that you can inspect the variable values or step through the code line by line. These are the basic usage of eclipse debugger. The following 7 tips will allow you do eclipse debug like a pro.

1. Change variable value
When a breakpoint is hit, you can right click a variable value then select "Change Value...".

2. Use breakpoint view
click Window -> Show View -> Breakpoint
In this tab, you can select/deselect breakpoints. High light one of the break point, you can set conditional match such as exceeding a certain hit count or matching a particular value.
For multi-threading program, "suspend VM" option allows you to exam the stack of the other threads. You can shift from one thread's stack to another thread's by clicking the thread name.

3. Use display view
click Window -> Show View -> Display
In this tab, you can write a few lines of code, highlight these code you want to run, then click the little triangle at the upper right corner to execute the code in the context of the breakpoint.

4. be able to inspecting code in project and depended jars.
If you are using maven project, install m2e plugin, then configure it to download both the source files and javadoc by going into Window > Preferences > Maven and checking the "Download Artifact Sources" and "Download Artifact JavaDoc" options.

You can search a class in project and depended jars with shortcut Command + Shift + T (mac) or Control + Shift + T (windows).
You can search a string in the workspace with shortcut Control + H.
You can search call hierarchy with shortcut Control + Command + H.

5. Remote debug
When you are trouble-shooting an application running in a remote machine. you need two piece of information: hostname or IP, the debugger port.
If you are not sure what port is the debugger port, you can login the hostname, then issue command
"netstat -nulpt | grep java " and see which ports the java application is listen to, then try these ports one by one.

6. set up tomcat to run local.
If your application produces a war file which can be deployed into tomcat, you can setup eclipse (J2EE) to integrate with tomcat and run your application local, then you can debug the code. To do so, download and unzip tomcat, in eclipse
Preference -> Server -> Runtime Environment
Click Add -> apache -> select the version matching the apache server you downloaded
Click next, browse to the directory of your downloaded apache server.
Click finish

Now you can run your j2EE project local by select Run -> Run As -> Run on Server
Click choose an existing server, then select your apache server from the server list.
Now tomcat will run your project's war file as a web container. Even with apache server configured, you program still need Arguments to be set correctly.

7. Use jrebel for quick deployment. JRebel fast tracks Java application development by skipping the time consuming build and redeploy steps in the development process.

7 unix commands for trouble-shooting memory related issue

Your application has been running fine for months until a recent linux patch are applied and you want to figure out what's going wrong.

These commands will be handy during your adventure of figuring out the root cause.

top

top command will give a general idea of how busy your system is and what resources are depleting. It will also list the top consumers of your computer cpu and memory. 

free -ht

free will list all the RAM usage statistics. -h is human readable, -t will print out total. The Total line is where you should find out the amount of free memory for your program. 

df -h

df -h let you check if you have disk full. -h is human readable.

uname -a

this command allow you to check the OS related information. From the output, you will find the kernel version, which may have bug related to performance issue.

lsof -nP | grep '(deleted)'

sometimes, you deleted a large file but your running application still have a reference to the file descriptor. The disk space won't be freed until the application exits. You can find these invisible files with command lsof -nP | grep '(deleted)'.

sar -r

when you want to find the memory usage in the past, sar command will do the job. The command sar have many flags for different resources like cpu, memory or IO, -r flag list the memory usage for today. 

ps -ef | grep java

find out if your java application is still running. To your surprise, it might just exit without any runtime exception in the log. The reason might be memory related.

When java program is running in VM that is configured to overcommit memory, the JVM will suffer from performance issue. A garbage collection which normally completes in hundreds of milliseconds might take minutes to complete in such an environment and the applications within that JVM are frozen during this time. 

“Java provides a special challenge in virtual environments due to the JVM’s introduction of a third level of memory management. The balloon driver draws memory from the virtual machine without impacting throughput because the guest OS efficiently claims pages that its processes are not using. But in the case of Java, the guest OS is unaware of how the JVM is using memory and is forced to select memory pages as arbitrarily and inefficiently as ESX’s swap routine. Neither ESX nor the guest OS can efficiently take memory from the JVM without significantly degrading performance. Memory in Java is managed inside the JVM and efforts by the host or guest to remove pages will both degrade Java applications’ performance. In these environments it is wise to manually set the JVM’s heap size and specify memory reservations for the virtual machine in ESX to account for the JVM, OS, and heap.“

7 steps to secure linux

Linux/Unix have many species, here we talk about those species designed for efficiency and security instead of luxury and convenience.

Nowadays, the only safe computer system is the dead brick. The one has no network connections -- wifi, bluetooth, cable, etc. You can only interact with it by physically sitting in front of it and typing on the keyboard.

step 1. get the trusted linux/unix distribution. 

Since most linux are open source, that means everybody have access to the source code and can modify it. You want to make sure you get the trusted distribution. So never download the Linux images from anywhere other than the official sources. Always be sure to verify the SHA256 checksums of the file you’ve downloaded against the official values. It would be easy for a malicious entity to modify a installation to contain exploits or malware and host it unofficially.

step 2. set a complex password for root

Without guarding root access, any security hardening is a waste of time.

step 3. boot into dead brick.

A runlevel is one of the modes that a Unix -based operating system will run in. Each runlevel has a certain number of services stopped or started, giving the user control over the behavior of the machine.

During the boot process for Redhat 9.0 and Fedora Core systems, for example, a sample /etc/inittab file defines the runlevel as follows:

# Default runlevel. The runlevels used by RHS are:
#   0 - halt (Do NOT set initdefault to this)
#   1 - Single user mode
#   2 - Multiuser, without NFS (The same as 3, if you do not have networking)
#   3 - Full multiuser mode
#   4 - unused
#   5 - X11
#   6 - reboot (Do NOT set initdefault to this)
#
id:2:initdefault:

This tells the init process that the default run level for the system is run level 2. This runlevel disables network access, solely use command line without the overhead of X11 based GUI.

step 4. disabling linux services

Now you have a safe dead brick, you can take time to disable any services that you don't actually need which expose extra access ports into your linux server if you leave them running in the background.

On redhat, to list all service settings run the following command:

    /sbin/chkconfig --list

This will display a long list of services showing whether or not they are started up at various runlevels. An example line looks like:

    httpd           0:off   1:off   2:off   3:on    4:off   5:on    6:off

chkconfig can also be used to change the settings. If we wanted the HTTP service not to start up when we at runlevel 5 we would issue the following command:

    /sbin/chkconfig --level 5 httpd off

on the other hand, you want to enable syslog
   /sbin/chkconfig --level 235 syslog on

step 5. set up firewall

Software firewall on your linux box is the second line of defense for your linux system, the main defense is the hardware firewall on your network gateway. So if you are in a dangerous environment, such as in shared public network, your software firewall is the only defense for your linux system.

The following command list all the firewall rules defined by your iptable:
iptables -L -v

You can start by the most restrict rule, then open some connection for the ones you know for sure.
The following rules deny all connections except a one way connection to ip 10.10.10.10 on port 80. HTTP connections TO 10.10.10.10 are permitted, but HTTP connections FROM 10.10.10.10 are not. However, the system is permitted to send back information over HTTP as long as the session has already been established.

iptables --policy INPUT DROP
iptables --policy OUTPUT DROP
iptables --policy FORWARD DROP
iptables -A OUTPUT -p http --dport ssh -s 10.10.10.10 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp --sport 80 -d 10.10.10.10 -m state --state ESTABLISHED -j ACCEPT

step 6. Update the OS

Keep the OS updated so that you get the latest security patches.

step 7. resist the temptation of installing unofficial applications

The single most common causes of a broken Linux installation are following unofficial advice, and particularly arbitrarily installing softwares from unofficial repositories.

ifconfig output on MacOs High Sierra

In a typical Mac OS X, type ifconfig in command line will give a long list of interfaces.

network>ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384

options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>

inet 127.0.0.1 netmask 0xff000000 

inet6 ::1 prefixlen 128 

inet6 xxx prefixlen 64 scopeid 0x1 

nd6 options=201<PERFORMNUD,DAD>

gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280

stf0: flags=0<> mtu 1280

EHC29: flags=0<> mtu 0

EHC26: flags=0<> mtu 0

XHC20: flags=0<> mtu 0

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

options=10b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV>

ether xxx 

nd6 options=201<PERFORMNUD,DAD>

media: autoselect (none)

status: inactive

en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

ether xxx 

inet6 xxx prefixlen 64 secured scopeid 0x8 

inet xxx netmask 0xffffff00 broadcast 192.168.2.255

nd6 options=201<PERFORMNUD,DAD>

media: autoselect

status: active

en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

options=60<TSO4,TSO6>

ether xxx 

media: autoselect <full-duplex>

status: inactive

fw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 4078

lladdr xxx 

nd6 options=201<PERFORMNUD,DAD>

media: autoselect <full-duplex>

status: inactive

p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304

ether xxx 

media: autoselect

status: inactive

awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484

ether xxx 

inet6 xxx%awdl0 prefixlen 64 scopeid 0xc 

nd6 options=201<PERFORMNUD,DAD>

media: autoselect

status: active

bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

options=63<RXCSUM,TXCSUM,TSO4,TSO6>

ether xxx 

Configuration:

id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0

maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200

root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0

ipfilter disabled flags 0x2

member: en2 flags=3<LEARNING,DISCOVER>

        ifmaxaddr 0 port 9 priority 0 path cost 0

nd6 options=201<PERFORMNUD,DAD>

media: <unknown type>

status: inactive

utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000

inet6 xxx%utun0 prefixlen 64 scopeid 0xe 

nd6 options=201<PERFORMNUD,DAD>

network>

The following are the explanations about these interfaces:


lo0 is the loopback device, which is used for entirely internal communication such as between two applications running on localhost.

gif0 The gif interface is a generic tunnelling device for IPv4 and IPv6. It can tunnel IPv[46] traffic over IPv[46]. Therefore, there can be four possible configurations. The behavior of gif is mainly based on RFC2893 IPv6-over-IPv4 configured tunnel. It is the mac os X default interface, not a security concern.

6to4 interfaces

stf0 is SixToFour interface. 6to4 is an Internet transition mechanism for migrating from Internet Protocol version 4 (IPv4) to version 6 (IPv6), a system that allows IPv6 packets to be transmitted over an IPv4 network (generally the IPv4 Internet) without the need to configure explicit tunnels. Special relay servers are also in place that allow 6to4 networks to communicate with native IPv6 networks. It is a default interface, not a security concern.

EHC29
EHC26
XHC20
With macOS High Sierra you can use Wireshark to capture USB traffic.  The interface needs to be manually brought up/down to enable/disable packet capture for the specific controller via ifconfig. The capture interfaces are named based on the underlying controller type followed by the bus number:

$ ifconfig
EHC26: flags=0<> mtu 0
XHC20: flags=0<> mtu 0
EHC29: flags=0<> mtu 0

$ ioreg -w0 -rc AppleUSBHostController
+-o XHC1@1400
+-o EHC2@1a00
+-o EHC1@1d00

The format above is @ where the most significant byte of the location is the bus number.  For example, if the device your interested in is connected to the XHCI controller XHC1@1400 then you would enable packet capture via "sudo ifconfig XHC20 up” and disable via “sudo ifconfig XHC20
down”.  Once the interface is up then Wireshark will be able to capture/decode/filter USB traffic for that controller.
These interfaces are MacOs High Sierra default, not security concern.

en0
en1
en2
physical network interfaces. Typically, one of them is the Ethernet interface, one of them is the Airport wifi interface, one of them is the bluetooth interface.

Firewire port

fw0 is networking over firewire. You can connect a Firewire cable between two Macs, and OS X can use that cable as a network connection.

Mac thunderbolt port

bridge0 is thunderbolt bridge. You can connect two Thunderbolt-equipped Mac computers using a Thunderbolt cable, then use internet protocol to communicate between the computers.

PPP (PPPSerial)

p2p0 peer to peer serial connection interfaces. If your MacOs have been connected to arduino through USB port, you will have this entry.

awdl0 AWDL (Apple Wireless Direct Link) is a low latency/high speed WiFi peer-to peer-connection Apple uses for everywhere you’d expect: AirDrop, GameKit (which also uses Bluetooth), AirPlay, and perhaps elsewhere. It works using its own dedicated network interface, typically “awdl0". By having multiple interfaces, Apple is able to have your standard WiFi connection on en*, while still broadcasting, browsing, and resolving peer to peer connections on awdl0.

oracle VirtualBox

utun0 it is the tunnel interface. If you have VirtualBox installed, guest operation system in virtualBox will use this interface to communicate with host operation system. TUN (namely network TUNnel) simulates a network layer device and it operates with layer 3 packets like IP packets. TUN is used with routing. Packets sent by an operating system via a TUN device are delivered to a user-space program which attaches itself to the device. A user-space program may also pass packets into a TUN device. In this case the TUN device delivers (or "injects") these packets to the operating-system network stack thus emulating their reception from an external source.

vment* - is used by VMWare Fusion to provide networking to your virtual machines, and there's likely to be one per VM you have set up.

ports and protocols on Mac OS X

mac os x use Darwin unix at the core, which is heavily influenced by BSD. The following is the ports.

network>cat /etc/protocols

#

# Internet protocols

#

# $FreeBSD$

# from: @(#)protocols 5.1 (Berkeley) 4/17/89

#

# See also http://www.iana.org/assignments/protocol-numbers

#

ip 0 IP # internet protocol, pseudo protocol number

#hopopt 0 HOPOPT # hop-by-hop options for ipv6

icmp 1 ICMP # internet control message protocol

igmp 2 IGMP # internet group management protocol

ggp 3 GGP # gateway-gateway protocol

ipencap 4 IP-ENCAP # IP encapsulated in IP (officially ``IP'')

st2 5 ST2 # ST2 datagram mode (RFC 1819) (officially ``ST'')

tcp 6 TCP # transmission control protocol

cbt 7 CBT # CBT, Tony Ballardie <A.Ballardie@cs.ucl.ac.uk>

egp 8 EGP # exterior gateway protocol

igp 9 IGP # any private interior gateway (Cisco: for IGRP)

bbn-rcc 10 BBN-RCC-MON # BBN RCC Monitoring

nvp 11 NVP-II # Network Voice Protocol

pup 12 PUP # PARC universal packet protocol

argus 13 ARGUS # ARGUS

emcon 14 EMCON # EMCON

xnet 15 XNET # Cross Net Debugger

chaos 16 CHAOS # Chaos

udp 17 UDP # user datagram protocol

mux 18 MUX # Multiplexing protocol

dcn 19 DCN-MEAS # DCN Measurement Subsystems

hmp 20 HMP # host monitoring protocol

prm 21 PRM # packet radio measurement protocol

xns-idp 22 XNS-IDP # Xerox NS IDP

trunk-1 23 TRUNK-1 # Trunk-1

trunk-2 24 TRUNK-2 # Trunk-2

leaf-1 25 LEAF-1 # Leaf-1

leaf-2 26 LEAF-2 # Leaf-2

rdp 27 RDP # "reliable datagram" protocol

irtp 28 IRTP # Internet Reliable Transaction Protocol

iso-tp4 29 ISO-TP4 # ISO Transport Protocol Class 4

netblt 30 NETBLT # Bulk Data Transfer Protocol

mfe-nsp 31 MFE-NSP # MFE Network Services Protocol

merit-inp 32 MERIT-INP # MERIT Internodal Protocol

dccp 33 DCCP # Datagram Congestion Control Protocol

3pc 34 3PC # Third Party Connect Protocol

idpr 35 IDPR # Inter-Domain Policy Routing Protocol

xtp 36 XTP # Xpress Tranfer Protocol

ddp 37 DDP # Datagram Delivery Protocol

idpr-cmtp 38 IDPR-CMTP # IDPR Control Message Transport Proto

tp++ 39 TP++ # TP++ Transport Protocol

il 40 IL # IL Transport Protocol

ipv6 41 IPV6 # ipv6

sdrp 42 SDRP # Source Demand Routing Protocol

ipv6-route 43 IPV6-ROUTE # routing header for ipv6

ipv6-frag 44 IPV6-FRAG # fragment header for ipv6

idrp 45 IDRP # Inter-Domain Routing Protocol

rsvp 46 RSVP # Resource ReSerVation Protocol

gre 47 GRE # Generic Routing Encapsulation

dsr 48 DSR # Dynamic Source Routing Protocol

bna 49 BNA # BNA

esp 50 ESP # encapsulating security payload

ah 51 AH # authentication header

i-nlsp 52 I-NLSP # Integrated Net Layer Security TUBA

swipe 53 SWIPE # IP with Encryption

narp 54 NARP # NBMA Address Resolution Protocol

mobile 55 MOBILE # IP Mobility

tlsp 56 TLSP # Transport Layer Security Protocol

skip 57 SKIP # SKIP

ipv6-icmp 58 IPV6-ICMP icmp6 # ICMP for IPv6

ipv6-nonxt 59 IPV6-NONXT # no next header for ipv6

ipv6-opts 60 IPV6-OPTS # destination options for ipv6

# 61 # any host internal protocol

cftp 62 CFTP # CFTP

# 63 # any local network

sat-expak 64 SAT-EXPAK # SATNET and Backroom EXPAK

kryptolan 65 KRYPTOLAN # Kryptolan

rvd 66 RVD # MIT Remote Virtual Disk Protocol

ippc 67 IPPC # Internet Pluribus Packet Core

# 68 # any distributed filesystem

sat-mon 69 SAT-MON # SATNET Monitoring

visa 70 VISA # VISA Protocol

ipcv 71 IPCV # Internet Packet Core Utility

cpnx 72 CPNX # Computer Protocol Network Executive

cphb 73 CPHB # Computer Protocol Heart Beat

wsn 74 WSN # Wang Span Network

pvp 75 PVP # Packet Video Protocol

br-sat-mon 76 BR-SAT-MON # Backroom SATNET Monitoring

sun-nd 77 SUN-ND # SUN ND PROTOCOL-Temporary

wb-mon 78 WB-MON # WIDEBAND Monitoring

wb-expak 79 WB-EXPAK # WIDEBAND EXPAK

iso-ip 80 ISO-IP # ISO Internet Protocol

vmtp 81 VMTP # Versatile Message Transport

secure-vmtp 82 SECURE-VMTP # SECURE-VMTP

vines 83 VINES # VINES

ttp 84 TTP # TTP

#iptm 84 IPTM # Protocol Internet Protocol Traffic

nsfnet-igp 85 NSFNET-IGP # NSFNET-IGP

dgp 86 DGP # Dissimilar Gateway Protocol

tcf 87 TCF # TCF

eigrp 88 EIGRP # Enhanced Interior Routing Protocol (Cisco)

ospf 89 OSPFIGP # Open Shortest Path First IGP

sprite-rpc 90 Sprite-RPC # Sprite RPC Protocol

larp 91 LARP # Locus Address Resolution Protocol

mtp 92 MTP # Multicast Transport Protocol

ax.25 93 AX.25 # AX.25 Frames

ipip 94 IPIP # Yet Another IP encapsulation

micp 95 MICP # Mobile Internetworking Control Pro.

scc-sp 96 SCC-SP # Semaphore Communications Sec. Pro.

etherip 97 ETHERIP # Ethernet-within-IP Encapsulation

encap 98 ENCAP # Yet Another IP encapsulation

# 99 # any private encryption scheme

gmtp 100 GMTP # GMTP

ifmp 101 IFMP # Ipsilon Flow Management Protocol

pnni 102 PNNI # PNNI over IP

pim 103 PIM # Protocol Independent Multicast

aris 104 ARIS # ARIS

scps 105 SCPS # SCPS

qnx 106 QNX # QNX

a/n 107 A/N # Active Networks

ipcomp 108 IPComp # IP Payload Compression Protocol

snp 109 SNP # Sitara Networks Protocol

compaq-peer 110 Compaq-Peer # Compaq Peer Protocol

ipx-in-ip 111 IPX-in-IP # IPX in IP

carp 112 CARP vrrp # Common Address Redundancy Protocol

pgm 113 PGM # PGM Reliable Transport Protocol

# 114 # any 0-hop protocol

l2tp 115 L2TP # Layer Two Tunneling Protocol

ddx 116 DDX # D-II Data Exchange

iatp 117 IATP # Interactive Agent Transfer Protocol

stp 118 STP # Schedule Transfer Protocol

srp 119 SRP # SpectraLink Radio Protocol

uti 120 UTI # UTI

smp 121 SMP # Simple Message Protocol

sm 122 SM # SM

ptp 123 PTP # Performance Transparency Protocol

isis 124 ISIS # ISIS over IPv4

fire 125 FIRE

crtp 126 CRTP # Combat Radio Transport Protocol

crudp 127 CRUDP # Combat Radio User Datagram

sscopmce 128 SSCOPMCE

iplt 129 IPLT

sps 130 SPS # Secure Packet Shield

pipe 131 PIPE # Private IP Encapsulation within IP

sctp 132 SCTP # Stream Control Transmission Protocol

fc 133 FC # Fibre Channel

rsvp-e2e-ignore 134 RSVP-E2E-IGNORE # Aggregation of RSVP for IP reservations

mobility-header 135 Mobility-Header # Mobility Support in IPv6

udplite 136 UDPLite # The UDP-Lite Protocol

mpls-in-ip 137 MPLS-IN-IP # Encapsulating MPLS in IP

manet 138 MANET # MANET Protocols (RFC5498)

hip 139 HIP # Host Identity Protocol (RFC5201)

shim6 140 SHIM6 # Shim6 Protocol (RFC5533)

wesp 141 WESP # Wrapped Encapsulating Security Payload (RFC5840)

rohc 142 ROHC # Robust Header Compression (RFC5858)

# 138-254 # Unassigned

pfsync 240 PFSYNC # PF Synchronization

# 253-254 # Use for experimentation and testing (RFC3692)

# 255 # Reserved

divert 258 DIVERT # Divert pseudo-protocol [non IANA]

network>