家财的个人 WIKI
梳理知识体系，记录编程生涯中踩过的坑与解决方案。
这里主要记录一些 copy & paste 的代码，有时候我们并不是需要了解某项技术的原理，需要的仅仅是能跑起来，用最精简的配置把整个流程能跑起来，这对于接触某些新项目是十分必要的，为了不要每次都重蹈覆辙，希望这里的笔记对大家也有帮助。
编程有时确实很无聊，各种环境搭建，OMG，干活去吧...
Install
- https://github.com/Kunde21/markdownfmt
go install github.com/Kunde21/markdownfmt@latest
- https://pandownload.com
MacOS
安装
brew cask install java
brew install maven
brew cask install intellij-idea-ce # IDE，不要告诉我你还在用 eclipse
可以通过执行/usr/libexec/java_home这个命令来获取JAVA_HOME
export JAVA_HOME="$(/usr/libexec/java_home)"
多版本管理
brew install jenv
cat ~/.bash_profile
JENV_ROOT="$HOME/.jenv"
if [ -d "${JENV_ROOT}" ]; then
export PATH="${JENV_ROOT}/bin:$PATH"
eval "$(jenv init -)"
fi
# 激活 export 插件来自动导出 JAVA_HOME 环境变量
jenv enable-plugin export
# 安装 jdk 到 jenv 中去
jenv add /Library/Java/JavaVirtualMachines/jdk-10.0.1.jdk/Contents/Home/
jenv add /Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/
jenv versions
- https://javastreets.com/blog/2017/9/using_multiple_java_sdk_versions.html
Hystrix
- https://github.com/Netflix/Hystrix/wiki/How-it-Works
创建
import com.netflix.hystrix.HystrixCommand;
import com.netflix.hystrix.HystrixCommandGroupKey;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class HelloWorldCommand extends HystrixCommand<String> {
private static final Logger logger = LoggerFactory.getLogger(HelloWorldCommand.class);
private final String name;
public HelloWorldCommand(String name) {
super(HystrixCommandGroupKey.Factory.asKey("default"));
this.name = name;
}
@Override
protected String run() throws Exception {
logger.info("HelloWorld Command Invoked");
return "Hello" + name;
}
}
使用
HelloWorldCommand helloWorldCommand = new HelloWorldCommand("World");
assertEquals("Hello World", helloWorldCommand.execute());
HelloWorldCommand helloWorldCommand = new HelloWorldCommand("World");
Future<string> future = helloWorldCommand.queue();
assertEquals("Hello World", future.get());
配置
hystrix.command.HelloWorldCommand.metrics.rollingStats.timeInMilliseconds=10000
hystrix.command.HelloWorldCommand.execution.isolation.strategy=THREAD
hystrix.command.HelloWorldCommand.execution.isolation.thread.timeoutInMilliseconds=1000
hystrix.command.HelloWorldCommand.execution.isolation.semaphore.maxConcurrentRequests=10
hystrix.command.HelloWorldCommand.circuitBreaker.errorThresholdPercentage=50
hystrix.command.HelloWorldCommand.circuitBreaker.requestVolumeThreshold=20
hystrix.command.HelloWorldCommand.circuitBreaker.sleepWindowInMilliseconds=5000
hystrix.threadpool.default.coreSize=10
hystrix.threadpool.default.queueSizeRejectionThreshold=5
参考
- https://dzone.com/articles/gentle-introduction-to-hystrix-hello-world
- https://github.com/Netflix/Hystrix/wiki/Configuration
Jvisualvm
连接 remote
在连接 remote 机器上的 Java 进程时，需要启动 jstatd 服务。
Jstatd
jstatd是一个rmi的server应用，用于监控jvm的创建和结束，并且提供接口让监控工具（如visualvm）可以远程连接到本机的jvms。注意是jvms，就是说运行jstatd命令后可以用监控工具监控本用户（运行jstatd命令的用户）所有已经启动的java程序。
如果直接运行 jstatd 命令，会报下面的错误：
[vagrant@vagrant-ubuntu-trusty-64 11:01:40 ~]
$ jstatd
Could not create remote object
access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")
java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.System.setProperty(System.java:792)
at sun.tools.jstatd.Jstatd.main(Jstatd.java:139)
这是由于没有指定 policy，可以用下面的命令启动：
jstatd -p 1099 -J-Djava.security.policy=<(echo 'grant codebase "file:${java.home}/../lib/tools.jar" {permission java.security.AllPermission;};') -J-Djava.rmi.server.hostname=192.168.33.10
值得注意的是需要指定 java.rmi.server.hostname，否则看不到 remote 机器上的进程
同理，启动 jmx 也需要指定，参考：
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=6789"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
export JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=192.168.33.10"
lein repl
参考
- http://clojure-goes-fast.com/blog/profiling-tool-jvisualvm/
- https://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstatd.html
- http://trinea.iteye.com/blog/1216917
- cannot start jstatd due to permission error
GC log
-verbose:gc
-Xloggc:/logs/gc-%t.log
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCCause
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=20M
-XX:+PrintFlagsFinal
-XX:+PrintTenuringDistribution
-XX:+PrintPromotionFailure
JVM内存调整参数
- https://github.com/jiacai2050/ideas/issues/40
OQL
SELECT * FROM java.lang.String s
WHERE ( s.count > 1000 ) = true
WHERE toString(s) = "monday"
WHERE dominators(s).size() = 0
WHERE s.@retainedHeapSize > 1024L
WHERE s.value != null AND s.value.@valueArray.@length >= 1 AND s.valu[email protected](0) = 'j'
WHERE s.@GCRootInfo != null
- https://help.eclipse.org/neon/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Freference%2Foqlsyntax.html
MacOS
安装
# 构建工具
brew install leiningen boot-clj
# Clojure installer and CLI tools
# https://clojure.org/guides/getting_started
brew install clojure
访问私有属性、方法
比如有一个 Java 类
public class ReflectDemo {
private final String hiddenField;
public ReflectDemo(String hiddenField) {
this.hiddenField = hiddenField;
}
public String getHiddenField() {
return hiddenField;
}
private String privateMethod(String name) {
return hiddenField + name;
}
}
问题：
- 如何在不调用 getHiddenField 方法的前提下访问 hiddenField
- 如何修改 final 的 hiddenField 字段
- 如何调用privateMethod 方法
以上三个问题的答案是：反射
;; 问题一
(let [demo (ReflectDemo. "haha")
hidden-field (.getDeclaredField ReflectDemo "hiddenField")]
(.setAccessible hidden-field true)
(= (.getHiddenField demo) (.get hidden-field demo))) ;; true
;; 问题二
(let [demo (ReflectDemo. "haha")
hidden-field (.getDeclaredField ReflectDemo "hiddenField")
modifiers-field (.getDeclaredField java.lang.reflect.Field "modifiers")]
(.setAccessible hidden-field true)
(.setAccessible modifiers-field true)
;; Remove "final" modifier
(.setInt modifiers-field hidden-field (bit-and (.getModifiers hidden-field)
(bit-not java.lang.reflect.Modifier/FINAL)))
(.set hidden-field demo "heihei")
(.get hidden-field demo)) ;; heihei
;; 问题三，这里需要注意的是 Java 里面的 varargs 在 Clojure 调用需要传入数组
(let [demo (ReflectDemo. "haha")
private-method (.getDeclaredMethod ReflectDemo "privateMethod" (into-array Class [String]))]
(.setAccessible private-method true)
(.invoke private-method demo (into-array String ["xxx"]))) ;; hahaxxx
Java 版本的解答
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
public class ReflectDemoMain {
public static void main(String[] args) throws Exception {
// 问题一
ReflectDemo demo = new ReflectDemo("haha");
Field hiddenField = ReflectDemo.class.getDeclaredField("hiddenField");
hiddenField.setAccessible(true);
System.out.println(hiddenField.get(demo));
// 问题二
Field modifiersField = Field.class.getDeclaredField("modifiers");
modifiersField.setAccessible(true);
modifiersField.setInt(hiddenField, hiddenField.getModifiers() & ~Modifier.FINAL);
hiddenField.set(demo, "heihei");
System.out.println(demo.getHiddenField());
// 问题三
Method privateMethod = ReflectDemo.class.getDeclaredMethod("privateMethod", String.class);
privateMethod.setAccessible(true);
System.out.println(privateMethod.invoke(demo, "xx"));
}
}
参考
- https://javahowtodoit.wordpress.com/2014/09/12/how-to-get-and-set-private-static-final-field-using-java-reflection/
概念理解
-
https://github.com/Day8/re-frame/blob/master/docs/API.md
-
Coeffect 类似 Ring 的 request map
-
Effect 类型 Ring 的 response map
-
Interceptors 类型 Ring 的 Middleware
Event Handler
;; for 90%
(reg-event-db
:my-event
(fn [db [_ bool]]
(dispatch [:do-something-else 3]) ;; oops, side-effect
(assoc db :send-spam new-val)))
;; cofx 输入 fx 输出
(reg-event-fx
:my-event
(fn [{:keys [db]} [_ a]]
{:db (assoc db :flag true)
:dispatch [:do-something-else 3]}))
Interceptors
{:id :something ;; decorative only
:before (fn [context] ...) ;; returns possibly modified context
:after (fn [context] ...)} ;; `identity` would be a noop
context
{:coeffects {:event [:some-id :some-param]
:db <original contents of app-db>}
:effects {:db <new value for app-db>
:dispatch [:an-event-id :param1]}
:queue <a collection of further interceptors>
:stack <a collection of interceptors already walked>}
Handler is also a interceptor
(defn db-handler->interceptor
[db-handler-fn]
(re-frame.core/->interceptor ;; a utility function supplied by re-frame
:id :db-handler ;; ids are decorative only
:before (fn [context]
(let [{:keys [db event]} (:coeffects context) ;; extract db and event from coeffects
new-db (db-handler-fn db event)] ;; call the handler
(assoc-in context [:effects :db] new-db)))))) ;; put db back into :effects
Effect Handler
(reg-fx
:db
(fn [value]
(reset! re-frame.db/app-db value)))
内置的 effect handler
- :db
{:db {:key1 value1 :key2 value2}} - :dispatch
{:dispatch [:event-id "param"] } - :dispatch-later
{:dispatch-later [{:ms 200 :dispatch [:event-id "param"]} {:ms 100 :dispatch [:also :this :in :100ms]}]} - :dispatch-n
- :deregister-event-handler
Coeffect Handler
(reg-event-fx
:some-id
[(inject-cofx :random-int 10) (inject-cofx :now) (inject-cofx :local-store "blah")] ;; 3
(fn [cofx _]
... in here I can access cofx's keys :now :local-store and :random-int))
(reg-cofx ;; registration function
:now ;; what cofx-id are we registering
(fn [coeffects _] ;; second parameter not used in this case
(assoc coeffects :now (js.Date.)))) ;; add :now key, with value
数据流图
figwheel
在 Clojure REPL 中启动 figwheel REPL
- 配置依赖
(defproject repler "0.1.0-SNAPSHOT"
...
:source-paths ["src"]
:profiles {:dev {:dependencies [[com.cemerick/piggieback "0.2.1"] <-- Note
[figwheel-sidecar "0.5.8"]]
:source-paths ["cljs_src"] }}
:repl-options {:nrepl-middleware [cemerick.piggieback/wrap-cljs-repl]} <-- Note
...
)
- 切换 REPL
(use 'figwheel-sidecar.repl-api)
(start-figwheel!) ;; 这里不会 merge profile，如果有必要可以手动设置
(def figwheel-config
{:figwheel-options {} ;; <-- figwheel server config goes here
:build-ids ["dev"] ;; <-- a vector of build ids to start autobuilding
:all-builds ;; <-- supply your build configs here
[{:id "dev"
:figwheel true
:source-paths ["cljs-src"]
:compiler {:main "repler.core"
:asset-path "js/out"
:output-to "resources/public/js/repler.js"
:output-dir "resources/public/js/out" }}]})
(start-figwheel! figwheel-config)
(cljs-repl)
参考：https://github.com/bhauman/lein-figwheel/wiki/Using-the-Figwheel-REPL-within-NRepl
Rust
国内加速
- https://mirror.tuna.tsinghua.edu.cn/help/rustup/
Go
module 相关教程
- 拜拜了，GOPATH君！新版本Golang的包管理入门教程
- https://github.com/golang/go/wiki/Modules#migrating-to-modules
- vendor 迁移到 module 时可能遇到的问题
MacOS
安装
推荐使用 nvm 进行多版本管理
brew install nvm
# 之后在 .bash_profile 中添加
export NVM_DIR="$HOME/.nvm"
[[ -s "$NVM_DIR" ]] && source "/usr/local/opt/nvm/nvm.sh"
nvm 常用命令
# 安装某个版本
nvm install node-${version}
# 查看本地安装的 node 版本
nvm ls
# 远端可安装的 long-term support 版的 node
nvm ls-remote --lts
# 把某个版本设为默认 node
nvm alias default <some-version>
# 迁移
nvm install node-${new-version} --reinstall-packages-from=node-${old-version}
# 卸载某个版本
nvm uninstall node-${version}
国内源加速
cat ~/.npmrc
registry=https://registry.npm.taobao.org/
disturl=https://npm.taobao.org/mirrors/node
ELECTRON_MIRROR=https://npm.taobao.org/mirrors/electron
安装
推荐使用 pyenv 进行多版本管理
brew install pyenv pyenv-virtualenv
# 之后在 .bash_profile 中添加
export PYENV_ROOT="${HOME}/.pyenv"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
pyenv 常用命令
pyenv 安装 python 时，会从源码 build，不同操作系统需要做些初始化工作，可参考：
# 安装 2.7.8
pyenv install 2.7.8
# 推荐选择国内源加速
v=3.7.1;wget http://mirrors.sohu.com/python/$v/Python-$v.tar.xz -P ~/.pyenv/cache/;
pyenv install $v
# 远端所有可安装的版本
pyenv install -l
# 查看本地安装的 python 版本
pyenv versions
# 设置当前目录的 python 版本，会在当前目录生成一个 .python-version 文件
pyenv local 2.7.6
# 设置全局默认版本，会写入 ~/.pyenv/version
pyenv global 2.7.6
pyenv-virtualenv 常用命令
# 用指定版本 python 生成 venv
pyenv virtualenv 2.7.10 my-virtual-env-2.7.10
# 用默认版本 生成 venv
pyenv virtualenv venv34
# 查看所有的 venv
pyenv virtualenvs
# 进入／退出 venv
pyenv activate <name>
pyenv deactivate
# 删除 venv
pyenv uninstall my-virtual-env
对于 10.14 版本之后的 macOS，在从源码 install 时会报下面的错误：
clang: error: linker command failed with exit code 1 (use -v to see invocation)
building '_tkinter' extension
clang -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/Users/liujiacai/.pyenv/versions/2.7.10/include -DWITH_APPINIT=1 -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tcl.framework/Versions/Current/PrivateHeaders -I/System/Library/Frameworks/Tk.framework/Headers -I/System/Library/Frameworks/Tk.framework/Versions/Current/PrivateHeaders -I/usr/X11R6/include -I/private/var/folders/hj/z2gkhrb17wvbytm6p79gmq1c0000gn/T/python-build.20190313145928.17581/Python-2.7.10/Mac/Include -I. -IInclude -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/Users/liujiacai/.pyenv/versions/2.7.10/include -I/usr/local/include -I/private/var/folders/hj/z2gkhrb17wvbytm6p79gmq1c0000gn/T/python-build.20190313145928.17581/Python-2.7.10/Include -I/private/var/folders/hj/z2gkhrb17wvbytm6p79gmq1c0000gn/T/python-build.20190313145928.17581/Python-2.7.10 -c /private/var/folders/hj/z2gkhrb17wvbytm6p79gmq1c0000gn/T/python-build.20190313145928.17581/Python-2.7.10/Modules/_tkinter.c -o build/temp.macosx-10.14-x86_64-2.7/private/var/folders/hj/z2gkhrb17wvbytm6p79gmq1c0000gn/T/python-build.20190313145928.17581/Python-2.7.10/Modules/_tkinter.o -framework Tk
clang: warning: -framework Tk: 'linker' input unused [-Wunused-command-line-argument]
In file included from /private/var/folders/hj/z2gkhrb17wvbytm6p79gmq1c0000gn/T/python-build.20190313145928.17581/Python-2.7.10/Modules/_tkinter.c:71:
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/tk.h:78:11: fatal error: 'X11/Xlib.h' file not found
# include <X11/Xlib.h>
^~~~~~~~~~~~
1 error generated.
Python build finished, but the necessary bits to build these modules were not found:
_bsddb _sqlite3 bsddb185
dbm dl imageop
linuxaudiodev nis ossaudiodev
spwd sunaudiodev zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
Failed to build these modules:
_Qt _tkinter
running build_scripts
running install_lib
这是由于 CTL 把一些头文件移动了位置，需要单独额外按照，方式如下：
sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /
- https://github.com/pyenv/pyenv/wiki/Common-build-problems
- https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes#3035624
安装
# https://github.com/pyenv/pyenv/wiki/Common-build-problems
sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bashrc
exec "$SHELL"
# 国内源下载 python
v=3.7.1;wget http://mirrors.sohu.com/python/$v/Python-$v.tar.xz -P ~/.pyenv/cache/;
pyenv install $v
pyenv install 2.7.16
git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash
MacOS
安装
推荐使用 rbenv 进行多版本管理
brew install rbenv
# 之后在 .bash_profile 中添加
export PATH="$PATH:$HOME/.rvm/bin"
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*
# 管理 gems 依赖
gem install bundler
rbenv 常用命令
# 远端所有可安装的版本
rbenv install -l
# 安装指定版本
rbenv install 2.0.0-p247
# 查看本地安装的 ruby 版本
rbenv versions
# 设置当前目录的 ruby 版本，会在当前目录生成一个 .ruby-version 文件
rbenv local 1.9.3-p327
# 取消当前目录默认 ruby 版本
rbenv local --unset
# 设置全局默认 ruby 版本
rbenv global 1.8.7-p352
pyenv-virtualenv 常用命令
# 用指定版本 python 生成 venv
pyenv virtualenv 2.7.10 my-virtual-env-2.7.10
# 用默认版本 生成 venv
pyenv virtualenv venv34
# 查看所有的 venv
pyenv virtualenvs
# 进入／退出 venv
pyenv activate <name>
pyenv deactivate
# 删除 venv
pyenv uninstall my-virtual-env
神之编辑器 Emacs
为了能在 git 中默认使用 Emacs，需要做以下配置：
# ~/.bashrc
export EDITOR="emacsclient -t -a=\"\""
export ALTERNATE_EDITOR=""
# ~/.gitconfig
[core]
editor = emacsclient -t -a=\\\"\\\"
上面的配置会调用 emacsclient 去连接 emacs daemon服务，如果服务没启，就先启动服务再去连接。
Emacs 配置文件推荐用 git 进行版本管理，方便迁移与交流。这是我的 .emacs.d。
入门教程
- https://www.gnu.org/software/emacs/tour/
- http://batsov.com/articles/2011/11/19/why-emacs/
常用命令
emacs --daemon=<session-name>
emacsclient -t -s <session-name>
alias e='emacsclient -t -a ""'
Org-mode 导出PDF文件
Org 导出 pdf 依赖于 LaTeX，Mac 下推荐安装 basictex 这个精简版的，可能需要单独使用 tlmgr 安装额外的 *.sty 文件
brew cask install basictex
sudo tlmgr update --self
sudo tlmgr install wrapfig
sudo tlmgr install capt-of
# PDF 导出时，需要 cjk 包，然后需要配置org-latex-packages-alist，具体参考 setup-org.el
sudo tlmgr install cjk
- https://orgmode.org/worg/org-tutorials/org-latex-export.html 有介绍如何改变导出引擎为 XeTeX
- https://orgmode.org/worg/org-dependencies.html
- https://orgmode.org/manual/LaTeX-header-and-sectioning.html
- https://www.pydanny.com/setting-up-latex-on-mac-os-x.html
- https://github.com/jiacai2050/dotfiles/blob/master/.emacs.d/customizations/setup-org.el
扩展阅读
- markdown 支持：https://github.com/yoshuawuyts/vmd
- GUI vs Teriminal
- https://www.emacswiki.org/emacs/iTerm2
- 终端使用 Emacs 与 Emacs GUI 的区别？
- https://www.gnu.org/software/emacs/manual/html_node/elisp/Images.html#Images
安装
# 安装命令
# org-mode 显示 inline image 时需要 imagemagick@6
brew tap d12frosted/emacs-plus
brew install emacs-plus
brew install sbcl # 顺便把 common lisp 也装上
# 以下是旧方法，brew 已经移除了编译选项
brew install emacs --with-cocoa --with-dbus --with-librsvg --with-imagemagick@6 --with-mailutils
- https://emacs.stackexchange.com/questions/30559/org-mode-inline-image-display-size/30560
- https://www.reddit.com/r/emacs/comments/bhjtf9/now_that_homebrew_has_removed_build_options_how/
(image-type-available-p 'imagemagick)
其实 Mac 是有自带 Emacs 的，位置是/usr/bin/emacs，只是版本非常旧，通过brew安装的位置在/usr/local/bin/emacs，可以通过下面的命令删除 Mac 自带的 Emacs：
sudo rm /usr/bin/emacs
sudo rm -rf /usr/share/emacs
键位映射
为了保证在远端服务器与本地开发的一致行，我希望在终端里面运行 emacs，而非 GUI 版本，但是这样也带来一个严重的问题，终端不支持一些特殊的键位组合，对于 Emacs 来说主要是一些 escape code，需要手动进行映射，解决方法可以参考 Stackoverflow 上的这个问题How do I bind C-= in emacs。
还有一点需要注意，Mac 自身有一些快捷键可能与 Emacs 冲突，主要是 Mission Control，我直接取消掉了。
如果上述方案无法解决你的问题，Google emacs + input-decode-map 关键字即可。Emacs China 也是一个好地方， 由于 Emacs 的使用门槛，决定了其社区的高质量，我的提问很快就得到了准确的解答。
或许我的配置文件也能给你一些启发：
(defun my/global-map-and-set-key (key command &optional prefix suffix)
"`my/map-key' KEY then `global-set-key' KEY with COMMAND.
PREFIX or SUFFIX can wrap the key when passing to `global-set-key'."
(my/map-key key)
(global-set-key (kbd (concat prefix key suffix)) command))
(defun my/map-key (key)
"Map KEY from escape sequence \"\e[emacs-KEY\."
(define-key function-key-map (concat "\e[emacs-" key) (kbd key)))
(global-set-key (kbd "C-c l") 'mc/edit-lines)
;; 需要配合 iTerm2 进行 key mapping
;; https://stackoverflow.com/a/40222318/2163429
(my/global-map-and-set-key "C-=" 'er/expand-region)
(my/global-map-and-set-key "C--" 'er/contract-region)
(my/global-map-and-set-key "C->" 'mc/mark-next-like-this)
(my/global-map-and-set-key "C-<" 'mc/mark-previous-like-this)
(my/global-map-and-set-key "C-c C->" 'mc/mark-all-like-this)
这里有一些参考文章：
- Fixing emacs bindings in iTerm2
- Where do I find a list of terminal key codes to remap shortcuts in bash?
国内下载镜像
- http://mirrors.ustc.edu.cn/gnu/emacs/
ELPA
- https://mirrors.tuna.tsinghua.edu.cn/help/elpa/
- https://elpa.emacs-china.org/
安装
apt 软件库里面 emacs 一般都比较旧，所以需要自己编译。
wget http://mirrors.ustc.edu.cn/gnu/emacs/emacs-25.3.tar.xz
tar xf emacs-25.3.tar.xz
# install build tools
sudo apt install build-essential checkinstall
./configure
make
make install
首先选择一个多 Python 环境管理工具，之前都是用的 virtualenv，最近尝试了 pyenv，相比之下功能更全面些，与 Node 的 nvm，Ruby 的 rvm 理念相同。
相关环境
- macOS Sierra 10.12.6
- Emacs 25.2.1
- pyenv
- pyenv-virtualenv
- elpy
- pyenv-mode
步骤
安装 pyenv pyenv-virtualenv
brew install pyenv pyenv-virtualenv
然后在~/.bashrc里面添加
export PYENV_ROOT="${HOME}/.pyenv"
if [ -d "${PYENV_ROOT}" ]; then
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
fi
然后使用 pyenv 来安装指定版本的 Python
pyenv install <some.version>
# 查看所以可选择安装的版本
pyenv install -l
# 使用 pyenv 管理 virtualenv
pyenv virtualenv <env-name>
# 设置全局默认 Python，可以指定虚拟环境或者某个版本
pyenv global <env-name or some.version>
# 在某一个项目下，固定某个环境或版本，它会在项目根目录下添加 `.python-version` 文件
pyenv local <env-name or some.version>
安装Emacs mode elpy pyenv-mode
elpy 安装稍微麻烦下，需要指定 archives
(add-to-list 'package-archives
'("elpy" . "https://jorgenschaefer.github.io/packages/"))
然后安装、初始化就可以了
M-x package-refresh-contents
M-x package-install RET elpy RET
;; 为了使用一些额外功能，需要安装一些 pip 包
pip install jedi flake8 importmagic autopep8 rope
;; 初始化
(package-initialize)
(elpy-enable)
这时 elpy 是不会使用 pyenv 指定的环境的，需要安装 pyenv-mode
M-x package-install RET pyenv-mode
# 指定虚拟环境或版本
M-x pyenv-mode-set
# 运行解释器
M-x run-python
参考
- https://smythp.com/emacs/python/2016/04/27/pyenv-elpy.html
- https://github.com/jorgenschaefer/elpy/wiki/Features
Docker
# 安装命令
brew cask install docker
国内访问 Docker Hub 有时会遇到困难，最好配置镜像加速器。
- http://registry.docker-cn.com
- https://mirror.tuna.tsinghua.edu.cn/help/docker-ce/
- https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/
- http://get.daocloud.io/
ENTRYPOINT vs. CMD
| No ENTRYPOINT | ENTRYPOINT exec_entry p1_entry | ENTRYPOINT [“exec_entry”, “p1_entry”] | |
|---|---|---|---|
| No CMD | error not allowed | /bin/sh -c exec_entry p1_entry | exec_entry p1_entry |
| CMD [“exec_cmd”, “p1_cmd”] | exec_cmd p1_cmd | /bin/sh -c exec_entry p1_entry | exec_entry p1_entry exec_cmd p1_cmd |
| CMD [“p1_cmd”, “p2_cmd”] | p1_cmd p2_cmd | /bin/sh -c exec_entry p1_entry | exec_entry p1_entry p1_cmd p2_cmd |
| CMD exec_cmd p1_cmd | /bin/sh -c exec_cmd p1_cmd | /bin/sh -c exec_entry p1_entry | exec_entry p1_entry /bin/sh -c exec_cmd p1_cmd |
常用命令
# 以交互式环境，执行一个 bash，并在 exit 后删除该 container
docker run -it --rm \
debian:14.04 \
bash
# 通过 volumns 在宿主机与 container 共享目录
docker run -d -v ~/nginxlogs:/var/log/nginx -p 5000:80 -i nginx
# 进入通过 -d 参数启动的后台 container
docker exec -it ${container_id} bash
# 虚悬镜像
docker images -f dangling=true
# 删除虚悬镜像
docker image prune
# 构建镜像，在具有 Dockerfile 的目录下执行
docker build -t nginx:v3 .
JVM in Docker and PTRACE_ATTACH
Docker 1.10 之后加了些安全限制，导致无法 ptrace 无法使用，进而导致 jmap 无法使用。解决访问
docker run --cap-add=SYS_PTRACE alpine sh -c 'apk add -U strace && strace echo'
## docker-compose 自1.1.0版本后支持 cap_add
version: '2'
services:
api:
...
cap_add:
- SYS_PTRACE
- https://jarekprzygodzki.wordpress.com/2016/12/19/jvm-in-docker-and-ptrace_attach/
- https://docs.docker.com/engine/reference/run/#additional-groups
Remote APIs
配置启动参数
DOCKER_OPTS="--registry-mirror=https://registry.docker-cn.com -H tcp://0.0.0.0:4243"
curl localhost:4243/images/json
- https://docs.docker.com/engine/api/v1.24/#3-endpoints
Dockerfile 示例
FROM java:7u121-jdk-alpine
MAINTAINER Jiacai Liu [email protected]
ENV KESTREL_VERSION 2.4.1
RUN set -xe \
\
&& wget -O kestrel.zip "http://twitter-archive.github.io/kestrel/download/kestrel-$KESTREL_VERSION.zip" \
&& mkdir /opt \
&& ls -lh \
&& unzip kestrel.zip -d /opt \
&& rm kestrel.zip \
&& ln -s /opt/kestrel-$KESTREL_VERSION /opt/kestrel
ADD ./config.scala /opt/kestrel/config.scala
WORKDIR /opt/kestrel
EXPOSE 2222 2223 2229 22133
# CMD & ENTRYPOINT doesn't expand ENV, so CMD below doesn't work
# CMD ["/usr/bin/java", "-jar", "/opt/kestrel/kestrel_2.9.2-$KESTREL_VERSION.jar", "-f", "/opt/kestrel/config.scala"]
# See: https://github.com/moby/moby/issues/4783
CMD ["sh", "-c", "/usr/bin/java -jar /opt/kestrel/kestrel_2.9.2-$KESTREL_VERSION.jar -f /opt/kestrel/config.scala"]
# 用 supervisor 启动多个命令
# CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf", "--nodaemon"]
安装
curl -L https://get.daocloud.io/docker/compose/releases/download/1.17.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors":["docker.mirrors.ustc.edu.cn"]
}
EOF
apt-get update
sudo apt-get install docker-ce
Virtualbox
# 安装命令
brew cask install virtualbox
在天朝，很多网站是只支持 IE 的，Mac 下的 Firefox, Chrome, Safari 这时候都显得心有力而不足了，而且很多软件也只有 Windows 版，所以装个虚拟机是非常有必要的。 Virtualbox 是我自用 Ubuntu 以来一直用的虚拟机，开源免费。
安装
Vagrant 是一款非常简单且实用的虚拟机命令行工具，支持市面上主流虚拟机，当然包括 VirtualBox，安装方式如下：
brew cask install vagrant # 虚拟机管理工具，方便命令行操作
为了介绍其使用，需要先了解两个名次：
- provider，vagrant 是一个通用的虚拟机管理工具，不仅仅是 virtualbox，还可以是 vmare aws，这些统称为 provider
- boxes，为了避免每次都是从头开始创建虚拟机，vagrant 提供 box 作为基本构件，可以根据需要构建自己的box。类似于 docker 里面的 image
常用命令
了解了上面两个概念，下面看看基于用法：
cd your-project
vagrant init hashicorp/precise64 # 安装 64 位的 Ubuntu 12.04
vagrant up # 启动
vagrant ssh # 进入 guest
# 退出 guest 后，可选择
vagrant suspend # 休眠
vagrant halt # 关机
vagrant destroy --force # 销毁该 guest
# 增加国内源 box
vagrant box add https://mirrors.tuna.tsinghua.edu.cn/ubuntu-cloud-images/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box --name tsing/trusty
vagrant box add https://mirrors.tuna.tsinghua.edu.cn/ubuntu-cloud-images/bionic/current/bionic-server-cloudimg-amd64-vagrant.box --name tsing/bionic
注意事项
磁盘大小
Vagrant 管理的 guest 默认系统磁盘一般是 10G，20G，如果觉得比较小，可以试试下面这个插件：
- https://github.com/sprotheroe/vagrant-disksize
我在 guest 为 debian 8 下测试没成功，大家可以试试运气。其他方法：
- https://gist.github.com/christopher-hopper/9755310
- https://medium.com/@phirschybar/resize-your-vagrant-virtualbox-disk-3c0fbc607817
- https://askubuntu.com/questions/317338/how-can-i-increase-disk-size-on-a-vagrant-vm/318711
同步目录
Vagrant 在启动时，会默认把当前目录挂载（mount）在 guest 的 /vagrant 目录，挂的方式有这么几种：nfs，rsync，smb，virtualbox，vagrant 会自动选择最佳的方式。如果 provider 是 Virtualbox，它会优先选择 virtualbox 共享目录的方式，但这里需要 box 提前安装了 VirtualBox Guest Additions，否则它会用 rsync 的方式，如果当前文件较多，这里启动时会比较慢，甚至可能出错。
不过幸好，官方提供的 hashicorp/precise64 默认安装了，所以基本不用担心。如果使用其他的 box，可参考官方手动安装的文档，基本分为2步：
- 安装内核头文件
sudo apt-get install linux-headers-$(uname -r) build-essential dkms
这一步如果出现类似E: Unable to locate package linux-headers-3.16.0-4-amd64的错误，说明当前源内没有当前系统的头文件，需要自行去 Google 找，Debian 的在这里
- 安装 Additions
wget http://download.virtualbox.org/virtualbox/5.1.26/VBoxGuestAdditions_5.1.26.iso
sudo mkdir /media/VBoxGuestAdditions
sudo mount -o loop,ro VBoxGuestAdditions_5.1.26.iso /media/VBoxGuestAdditions
sudo sh /media/VBoxGuestAdditions/VBoxLinuxAdditions.run
rm VBoxGuestAdditions_5.1.26.iso
sudo umount /media/VBoxGuestAdditions
sudo rmdir /media/VBoxGuestAdditions
示例用的是5.1.26版本，最好与本地安装的 Virtualbox 版本一致。
为了方便今后操作，最好把安装好 Additions 的 box 打包一下，方便以后使用，方法也很简单：
vagrant package --output mybox.box
# 安装到本地
vagrant box add mybox mybox.box
# 使用
vagrant init mybox
最后，可以发布到 Vagrant Cloud 方便不同机器复用。我自己制作了一个基于 debian8 的 box，安装上了 Clojure 开发环境，一键即可安装。
Vagrantfile for Ubuntu 18.04 推荐使用
# coding: utf-8
# -*- mode: ruby -*-
# vi: set ft=ruby :
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
# The most common configuration options are documented and commented below.
# For a complete reference, please see the online documentation at
# https://docs.vagrantup.com.
# Every Vagrant development environment requires a box. You can search for
# boxes at https://vagrantcloud.com/search.
# config.vm.box = "ubuntu/trusty64"
config.vm.box = "tsing/bionic"
config.disksize.size = '100GB'
# Disable automatic box update checking. If you disable this, then
# boxes will only be checked for updates when the user runs
# `vagrant box outdated`. This is not recommended.
# config.vm.box_check_update = false
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
# NOTE: This will enable public access to the opened port
# config.vm.network "forwarded_port", guest: 80, host: 8080
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine and only allow access
# via 127.0.0.1 to disable public access
# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Create a private network, which allows host-only access to the machine
# using a specific IP.
config.vm.network "private_network", ip: "192.168.33.10"
# Create a public network, which generally matched to bridged network.
# Bridged networks make the machine appear as another physical device on
# your network.
# config.vm.network "public_network"
# Share an additional folder to the guest VM. The first argument is
# the path on the host to the actual folder. The second argument is
# the path on the guest to mount the folder. And the optional third
# argument is a set of non-required options.
# config.vm.synced_folder "../data", "/vagrant_data"
config.vm.synced_folder ".", "/vagrant"
config.vm.synced_folder "~/.m2", "/home/vagrant/.m2"
config.vm.synced_folder "~/.lein", "/home/vagrant/.lein"
# config.vm.synced_folder "~/.emacs.d/", "/home/vagrant/.emacs.d"
# Provider-specific configuration so you can fine-tune various
# backing providers for Vagrant. These expose provider-specific options.
# Example for VirtualBox:
#
config.vm.provider "virtualbox" do |vb|
# Display the VirtualBox GUI when booting the machine
vb.gui = false
vb.memory = "8192"
vb.cpus = 2
end
#
# View the documentation for the provider you are using for more
# information on available options.
# Enable provisioning with a shell script. Additional provisioners such as
# Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
# documentation for more information about their specific syntax and use.
config.vm.provision "file", source: "~/.gitconfig", destination: ".gitconfig"
config.vm.provision "file", source: "~/.tmux.conf", destination: ".tmux.conf"
config.vm.provision "file", source: "~/.bashrc", destination: ".bash_aliases"
# config.vm.provision "file", source: "./lein", destination: "lein"
config.vm.provision "shell", inline: <<-SHELL
sudo cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
cat > /etc/apt/sources.list << EOF
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
# 预发布软件源，不建议启用
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
EOF
apt-get update
apt-get -y install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
cat >> ~/.bash_aliases << EOF
alias docker='sudo docker'
alias docker-compose='sudo docker-compose'
alias dc='sudo docker-compose'
EOF
apt-get update
apt-get -y install emacs docker-ce
cat >> /etc/default/docker << EOF
DOCKER_OPTS="--registry-mirror=https://registry.docker-cn.com"
EOF
service docker restart
JAVA_SHA256_SUM=6d34ae147fc5564c07b913b467de1411c795e290356538f22502f28b76a323c2
JAVA_HOME=/opt/jdk/jdk1.8.0_192
mkdir -p /opt/jdk
cd /opt
wget http://lc-lhzo7z96.cn-n1.lcfile.com/1551684078102 -O java.tar.gz
echo "$JAVA_SHA256_SUM java.tar.gz" | sha256sum -c -
tar -zxf java.tar.gz -C /opt/jdk
cd $JAVA_HOME/bin
for cmd in *; do
sudo update-alternatives --install /usr/bin/$cmd $cmd $JAVA_HOME/bin/$cmd 100;
done
rm -rf /opt/java.tar.gz
SHELL
end
Vagrantfile for Debian 8
# -*- mode: ruby -*-
# vi: set ft=ruby :
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
# The most common configuration options are documented and commented below.
# For a complete reference, please see the online documentation at
# https://docs.vagrantup.com.
# Every Vagrant development environment requires a box. You can search for
# boxes at https://vagrantcloud.com/search.
config.vm.box = "my-box"
# config.disksize.size = '50GB'
# Disable automatic box update checking. If you disable this, then
# boxes will only be checked for updates when the user runs
# `vagrant box outdated`. This is not recommended.
# config.vm.box_check_update = false
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
# NOTE: This will enable public access to the opened port
# config.vm.network "forwarded_port", guest: 80, host: 8080
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine and only allow access
# via 127.0.0.1 to disable public access
# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
# Create a private network, which allows host-only access to the machine
# using a specific IP.
config.vm.network "private_network", ip: "192.168.33.10"
# Create a public network, which generally matched to bridged network.
# Bridged networks make the machine appear as another physical device on
# your network.
# config.vm.network "public_network"
# Share an additional folder to the guest VM. The first argument is
# the path on the host to the actual folder. The second argument is
# the path on the guest to mount the folder. And the optional third
# argument is a set of non-required options.
# config.vm.synced_folder "../data", "/vagrant_data"
config.vm.synced_folder ".", "/vagrant"
config.vm.synced_folder "~/.m2", "/home/vagrant/.m2"
config.vm.synced_folder "~/.lein", "/home/vagrant/.lein"
# config.vm.synced_folder "~/.emacs.d/", "/home/vagrant/.emacs.d"
# Provider-specific configuration so you can fine-tune various
# backing providers for Vagrant. These expose provider-specific options.
# Example for VirtualBox:
#
config.vm.provider "virtualbox" do |vb|
# Display the VirtualBox GUI when booting the machine
vb.gui = false
vb.memory = "4096"
end
#
# View the documentation for the provider you are using for more
# information on available options.
# Enable provisioning with a shell script. Additional provisioners such as
# Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
# documentation for more information about their specific syntax and use.
config.vm.provision "file", source: "~/.gitconfig", destination: ".gitconfig"
config.vm.provision "file", source: "~/.tmux.conf", destination: ".tmux.conf"
config.vm.provision "file", source: "~/.bashrc", destination: ".bash_aliases"
config.vm.provision "file", source: "./lein", destination: "lein"
config.vm.provision "shell", inline: <<-SHELL
mv lein /usr/local/bin/
sudo cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
curl -L https://get.daocloud.io/docker/compose/releases/download/1.17.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
cat > /etc/apt/sources.list << EOF
deb http://mirrors.163.com/debian/ jessie main non-free contrib
deb http://mirrors.163.com/debian/ jessie-updates main non-free contrib
deb http://mirrors.163.com/debian/ jessie-backports main non-free contrib
deb http://mirrors.163.com/debian-security/ jessie/updates main non-free contrib
deb [arch=amd64] https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/debian jessie stable
# deb-src http://mirrors.163.com/debian/ jessie main non-free contrib
# deb-src http://mirrors.163.com/debian/ jessie-updates main non-free contrib
# deb-src http://mirrors.163.com/debian/ jessie-backports main non-free contrib
# deb-src http://mirrors.163.com/debian-security/ jessie/updates main non-free contrib
EOF
apt-get update
apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors":["docker.mirrors.ustc.edu.cn"]
}
EOF
cat >> ~/.bash_aliases << EOF
alias docker='sudo docker'
alias docker-compose='sudo docker-compose'
alias dc='sudo docker-compose'
EOF
cat >> /home/vagrant/.bashrc <<EOF
export JAVA_HOME=/vagrant/jdk1.8.0_152
export PATH=$PATH:/vagrant/jdk1.8.0_152/bin
EOF
apt-get update
apt-get -y install emacs docker-ce
SHELL
end
抓包工具 Wireshark
# 安装命令
brew cask install wireshark
也许是最强大的抓包工具，从其名字上就能体现出：wire（线路）+ shark（鲨鱼）。但这个软件初次使用时有些难度，最重要的是区分两个概念：
capture filter，在抓包开始时指定。
常见表达式
# Capture only traffic to or from IP address 172.18.5.4:
host 172.18.5.4
# Capture traffic to or from a range of IP addresses:
net 192.168.0.0/24
# Capture non-HTTP and non-SMTP traffic on your server (both are equivalent):
host www.example.com and not (port 80 or port 25)
host www.example.com and not port 80 and not port 25
# Capture traffic within a range of ports with newer versions of libpcap (0.9.1 and later):
tcp portrange 1501-1549
#Capture only IPv4 traffic - the shortest filter, but sometimes very useful to get rid of lower layer protocols like ARP and STP:
ip
# Capture only unicast traffic - useful to get rid of noise on the network if you only want to see traffic to and from your machine, not, for example, broadcast and multicast announcements:
not broadcast and not multicast
display filter，在抓取一定包后进行过滤。
常见表达式
ip.dst_host == 192.168.30.103 and tcp.dstport == 80
ip.addr == 10.43.54.65
# is equivalent to
ip.src == 10.43.54.65 or ip.dst == 10.43.54.65
- Getting a Packet Trace
- Capture a packet trace using Terminal on your Mac
- https://jvns.ca/blog/2018/06/19/what-i-use-wireshark-for/
配置
ControlMaster
对于 OpenSSH 4.0 以及之后的版本，引入一新功能 ControlMaster，可以复用之前已经登录的连接，建议开启：
Host *
ControlMaster auto
ControlPath ~/.ssh/master-%r@%h:%p
ControlPersist 60m
参考：
- Accelerating OpenSSH connections with ControlMaster
- https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing
临时禁用公钥登录：
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no example.com
MacOS
在 macOS/OS X 截止到 Yosemite，ssh-agent 会一直记住 ssh-add -K 添加的 key，即使重启 keychain，ssh-agent 也会自动去读取保存在 keychain 中的密码（passphrase）。但在 Sierra 中，重启 keychain 后，ssh-agent 就不会去读取了。Apple 开发者也对这一现象作出回应：
That’s expected. We re-aligned our behavior with the mainstream OpenSSH in this area.
解决办法也很简单，将ssh-add -A 放到 ~/.bashrc 里面就可以了。除了这种方式，还可以在~/.ssh/config里面加入如下配置：
Host * (asterisk for all hosts or add specific host)
AddKeysToAgent yes
UseKeychain yes
IdentityFile <key> (e.g. ~/.ssh/userKey)
参考：
Ubuntu
sudo apt-get install keychain
# ~/.bashrc
/usr/bin/keychain $HOME/.ssh/id_ecdsa
source $HOME/.keychain/$HOSTNAME-sh
- https://www.cyberciti.biz/faq/ubuntu-debian-linux-server-install-keychain-apt-get-command/
pssh
pssh -h hs -A -i -l root "docker ps"
# 对于执行时间较长对命令，可以用 -t 指定超时秒数，默认 60s
验证 keep-alive
curl -Iv http://baidu.com 2>&1 | grep -i 'connection #0'
* Connection #0 to host baidu.com left intact
Connection ... left intact 说明 server 没有关闭连接，等待 client 重用。client 决定是否复用连接。
curl -Iv http://baidu.com --next http://baidu.com 2>&1 | grep -i '#0'
* Connected to baidu.com (39.156.69.79) port 80 (#0)
* Connection #0 to host baidu.com left intact
* Re-using existing connection! (#0) with host baidu.com
* Connected to baidu.com (39.156.69.79) port 80 (#0)
* Connection #0 to host baidu.com left intact
- https://serverfault.com/a/554668/252786
top
top
Once in top...
-
P <- Sort by CPU usage
-
M <- Sort by MEM usage
-
z <- Add cool visual colors
-
x <- Highlight column you are currently sorting by
-
H <- 显示线程
printf '%x\n' 26
可以把十进制的 26 转为十六进制的 1a
ps
ps -p <pid> -o sz,rss,vsz
- sz, 进程映像所占用的物理页面数量，也就是以物理页面为单位表示的虚拟内存大小
- rss, resident set size, the non-swapped physical memory that a task has used (in kiloBytes). (alias rssize, rsz).
- vsz, 进程的虚拟内存大小
# sz and vsz represent the same thing, but sz is in page units, while vsz is in 1024 byte units.
# 可以拿到物理页大小
getconf PAGE_SIZE
vsz * 1024 = sz * page_size
rss <= vsz
查看网速
ethtool eth0 | grep -i Speed
Speed: 1000Mb/s
apt install sysstat
sar -n DEV 1
11:48:54 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
11:48:54 eth0 17.07 12.20 1.23 1.86 0.00 0.00 0.00
11:48:54 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
Average: eth0 20.91 22.80 3.67 18.55 0.00 0.00 0.00
Average: lo 1.49 1.49 0.39 0.39 0.00 0.00 0.00
ss
Another utility to investigate sockets
基本用法
ss -t -a
Display all TCP sockets.
ss -u -a
Display all UDP sockets.
ss -o state established '( dport = :ssh or sport = :ssh )'
Display all established ssh connections.
ss -x src /tmp/.X11-unix/*
Find all local processes connected to X server.
ss -o state fin-wait-1 '( sport = :http or sport = :https )' dst 193.233.7/24
List all the tcp sockets in state FIN-WAIT-1 for our apache to network 193.233.7/24 and look at their timers.
ss -i 详解
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 ::ffff:127.0.0.1:2181 ::ffff:127.0.0.1:52150
cubic wscale:6,7 rto:204 rtt:7.5/3 ato:40 mss:21888 cwnd:10 ssthresh:7 send 233.5Mbps rcv_rtt:22200.7 rcv_space:43690
- Recv-Q，The count of bytes not copied by the user program connected to this socket.
- Send-Q，The count of bytes not acknowledged by the remote host.
- cubic，TCP 拥塞控制使用的算法，通过
/proc/sys/net/ipv4/tcp_congestion_control设置 - wscale，rwnd 的 scale 值，逗号前面的 6 对远端提供的 rwnd 起作用，逗号后面的 7 对我们提供给远端的 rwnd 起作用。
/proc/sys/net/ipv4/tcp_window_scaling控制是否开启 scale - rto，Retransmittion Timeout，204ms 内没收到 ack，理论上最小为1s，但 Linux 实现中默认一般小于1s，见参考5。
netstat -s | grep retrans可以看到重传数 - rtt，Round Trip Time (is the delays between sent packets and received ACK)
- cwnd，current congestion window size
- ssthresh，a system-configured congestion threshold。在 tcp 慢启动阶段，cwnd <= ssthresh 。在
/sys/module/tcp_cubic/parameters/initial_ssthresh中设置，默认为 0. - rcv_rtt，The time to receive one full window
- rcv_space，twice the most recently measured window? TCP's internal auto-tuning to grow socket buffers based on how much data the kernel estimates the sender can send
参考：
- http://www.misterx.org/wiki/index.php/Ss
- https://github.com/jiacai2050/ideas/issues/41#issuecomment-362791333
- What is rcv_space in the 'ss --info' output, and why it's value is larger than net.core.rmem_max
- https://blog.janestreet.com/inspecting-internal-tcp-state-on-linux/
- Performance Best Practice: Monitor TCP Retransmits
- http://www.lognormal.com/blog/2012/09/27/linux-tcpip-tuning/
ss 源码
- https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/tree/misc/ss.c
使用 https://shadowsocks.org/ 进行科学上网
- https://shadowsocks.org/en/config/quick-guide.html
Server
推荐去 Google Compute Engine 购买虚拟机，有一年的免费额度。 如果自己有域名，可以绑定一个二级域名的 A 记录到虚拟机的外网 IP，方便记忆。
# 1. 安装 ss 服务端
pip install shadowsocks
# 2. 编辑配置文件
# https://wiki.archlinux.org/index.php/Shadowsocks
cat /etc/shadowsocks.json
{
"server":"0.0.0.0",
"server_port": 443,
"password":"mypassword",
"timeout":600,
"method":"aes-256-cfb",
"fast_open": true,
"workers": 2
}
# 3. 安装 tmux 多屏服用
yum install -y tmux
# 4. nohup 启动, 最好在 tmux 内执行
nohup ss-server -c /etc/shadowsocks.json
Client
brew install shadowsocks-libev
Mac 下不推荐安装 GUI 版本，已经很久没人维护了。安装之后编辑/usr/local/etc/shadowsocks-libev.json，填入 server 地址即可。
cat /usr/local/etc/shadowsocks-libev.json
{
"server": "your-server-ip-or-domain",
"server_port": 443,
"local_port": 1080,
"password": "mypassword",
"timeout": 600,
"method": "aes-256-cfb"
}
# 测试
ss-local -v -c /usr/local/etc/shadowsocks-libev.json
# 开机启动
brew services start shadowsocks-libev
为了让终端可以使用代理，需要将 http(s) 转为 socks 流量。ss 官方推荐的是 proxychains，但是在OS X 10.11 以后引入了 SIP安全机制，导致无法直接使用，关闭 SIP 貌似也不可取，可以选用 privoxy 来替代 proxychains。（参考）
brew install privoxy
# privoxy 使用 8118 端口， ss 使用 1080
echo 'listen-address 0.0.0.0:8118\nforward-socks5 / localhost:1080 .' >> /usr/local/etc/privoxy/config
# 测试，查看 8118 有没有在监听， netstat -an | grep 8118
/usr/local/sbin/privoxy /usr/local/etc/privoxy/config
# 开机启动
brew services start privoxy
经过上面这几步 http(s)->socks5 就完成，下面只需要让终端走这个代理即可：
export http_proxy='http://localhost:8118'
export https_proxy='http://localhost:8118'
# 可以将以下函数放入 ~/.bashrc 中，方便开启/关闭代理
function proxy_off(){
unset http_proxy
unset https_proxy
echo -e "已关闭代理"
}
function proxy_on() {
export no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"
export http_proxy="http://127.0.0.1:8118"
export https_proxy=$http_proxy
echo -e "已开启代理"
}
Ubuntu
sudo apt-get install shadowsocks-libev privoxy
echo 'forward-socks5 / localhost:1080 .' >> /etc/privoxy/config
sudo systemctl restart privoxy
# 前台启动
ss-local -v -c shadowsocks-libev.json
安装
sudo apt-get install -y sysstat
## 安装后直接运行会报错
## Cannot open /var/log/sysstat/sa04: No such file or directory
## Please check if data collecting is enabled in /etc/default/sysstat
## 修改 ENABLED="true"
sudo vi /etc/default/sysstat
# 重启服务，sar 就可以使用了
sudo service sysstat restart
参考
- http://linuxtools-rst.readthedocs.io/zh_CN/latest/tool/sar.html
安装
sudo apt-get update && apt-get install -y supervisor
# for centos
yum install -y python-setuptools
easy_install supervisor
supervisord.conf
;; 相关目录必须提前创建
[unix_http_server]
file=/var/run/supervisor/supervisor.sock ; (the path to the socket file)
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
user=root
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = supervisord.d/*.ini
program
[program:cse]
command=/opt/ceresdb/cse-server -c /opt/ceresdb/ecs-2c4g.toml
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
numprocs=1 ; number of processes copies to start (def 1)
umask=022 ; umask for process (default None)
priority=999 ; the relative start priority (default 999)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=10 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=TERM ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/opt/ceresdb/logs/cse_out.log
stdout_logfile_maxbytes=50MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=5 ; # of stdout logfile backups (default 10)
stderr_logfile=/opt/ceresdb/logs/cse_err.log
stderr_logfile_maxbytes=50MB ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=5 ; # of stderr logfile backups (default 10)
environment=A=1,B=2 ; process environment additions (def no adds)
serverurl=AUTO
终端 iTerm 2
# 安装命令
brew cask install iterm2
Mac自带的终端不是很强，程序员们需要一个强劲的终端来工作，于是有了 iTerm2。需要设置一项：Left Option act as +Esc（Preferences-->Profiles-->Keys），这样 Option 就可以用作 Meta 键了。
| 快捷键 | 功能 |
|---|---|
| F11 | 查看桌面 |
| F12 | 查看Dashboard |
| Cmd + <- | HOME |
| Cmd + -> | END |
| Shift + Control + 电源键 | 锁屏 （Windows 下为Win+L） |
| Cmd + Option + esc | 强制关闭程序 （Windows 下为Ctrl+Alt+Delete） |
| Cmd + ~ | 在同一应用不同窗口切换 |
| Cmd + Option + V | Finder 里面剪切 |
| Cmd + D | 垂直分屏 |
| Cmd + Shift + D | 水平分屏 |
| Cmd + T | 开多个Tab |
| Cmd + 数字 | 在多个Tab之间进行切换 |
| Ctrl + r | 搜索历史命令 |
| !! | 执行上条命令 |
| Option + F/B | 向前、向后按字移动。Bash Shell 光标默认按照 Emacs 风格移动，也可改为 VIM，可参考Modifying the Bash Shell with the set Command。 |
| Ctrl+x Ctrl+e | 调用默认编辑器去编辑一个特别长的命令 |
当然，说到了 iTerm2，不得不提到终端复用软件 tmux，tmux 默认配置文件在 Mac 上很别扭，你可以参考我这里的配置文件，这样 tmux 就可以像 vim 一样实现各种分屏的效果了。如果你还不知道 tmux 为何物，强烈推荐你看这个13分钟的视频，绝对物超所值，感谢 happypeter 的分享。
我现在用的主题是：Tomorrow Night。
~/.bashrc
Linux 一般通过~/.bashrc进行环境变量的配置，但是在 Mac 下配置后，发现根本没有效果，这是为什么呢？
其实这是个比较基础的问题，shell有两种：登录式shell与非登录式shell，直观理解，登录(login)式shell就是在你打开shell要求你输入用户名与密码的shell，我们在使用桌面Linux时一般只在登录时接触到这种shell，等我们进入系统后，再打开的Terminal就是非登录式shell了。这两种 shell 读取配置的文件是不同的：
- 登录式Shell启动时会去读取
~/.profile文件（Redhat、Mac上为~/.bash_profile） - 非登录式Shell会去读取
~/.bashrc文件
这也就解释了为什么在 Linux 系统上只需要修改 ~/.bashrc 后即可生效的原因。但在 Mac 上有些不同，开机后再通过 Terminal.app（或iTerm.app） 打开终端时，这时的 shell 是登录式shell，因为Terminal.app（或iTerm.app）这个应用程序是通过/usr/bin/login这个命令打开终端的，所以不会去source ~/.bashrc了。
解决方法也很简单，在~/.bash_profile加上下面一句代码就ok了
[ -r ~/.bashrc ] && source ~/.bashrc
Mac下ls命令默认是没有颜色的，不是很直观，可以自己设置一个alias，参考链接
export LSCOLORS=gxBxhxDxfxhxhxhxhxcxcx
alias ls='ls -FG'
alias ll='ls -l'
为了在命令行提示符中显示时间，可以设置PS1变量
export PS1="\n\e[1;37m[\e[m\e[1;35m\u\e[m\e[1;36m@\e[m\e[1;37m\h\e[m \e[1;33m\t\e[m \w\e[m\e[1;37m]\e[m\e[1;36m\e[m\n\$ "
# 效果如下
[liujiacai@macbook 22:02:13 ~]
一些实用命令
# 修改hostname
sudo scutil --set HostName <name>
# 查看USB设备
system_profiler SPUSBDataType
下面的命令需要通过brew进行安装后在使用
# 查看网络请求
brew install httpstat
$ httpstat baidu.com
Connected to 180.149.132.47:80 from 172.17.10.80:54727
HTTP/1.1 200 OK
Date: Sat, 14 Jan 2017 13:49:16 GMT
Server: Apache
Last-Modified: Tue, 12 Jan 2010 13:48:00 GMT
ETag: "51-47cf7e6ee8400"
Accept-Ranges: bytes
Content-Length: 81
Cache-Control: max-age=86400
Expires: Sun, 15 Jan 2017 13:49:16 GMT
Connection: Keep-Alive
Content-Type: text/html
Body stored in: /var/folders/2g/fxz_98ks0lgc79sjp5vn5cxc0000gn/T/tmpsawHq4
DNS Lookup TCP Connection Server Processing Content Transfer
[ 69ms | 37ms | 33ms | 0ms ]
| | | |
namelookup:69ms | | |
connect:106ms | |
starttransfer:139ms |
total:139ms
# Swiss Army Knife for macOS !
brew install m-cli
$ m trash status
Size: 51G
Number of files: 252172
折腾过无数编辑器，最终停留在了 Emacs，偶尔也会用 TextMate 编辑汉字，Emacs 下汉字有些卡。
TextMate
brew cask install textmate
Sublime/Atom/VSCode
# 安装命令
brew cask install sublime-text
brew cask install atom
brew cask install visual-studio-code
再早之前，依次用过 sublime -> atom -> vscode，最后还是回到了 sublime 的怀抱，我个人不是很喜欢用 JS 写的 vscode/atom，相比之下我更相信 C++ 写的 sublime，比较编辑器这东西对性能要求还是挺高的，尤其是知道了 sublime 是一个 Google 员工辞职开发的产品，这才是情怀好嘛！
由于 sublime 出现的最早，所以 atom 与 vscode 都提供了与之类似的快捷键：
下面是我非常依赖的快捷键：
- Multiple Selection:
Control+Command+G（在 Linux/Windows 下，是 Alt+F3） - 选中多行
Shift+Command+L - 多行合并为一行
Command+J
sublime 插件安装
首先是 Package Manager，推荐直接用 sublime text 3。ctrl+` 打开控制台，输入
import urllib.request,os,hashlib; h = '6f4c264a24d933ce70df5dedcf1dcaee' + 'ebe013ee18cced0ef93d5f746d80ef60'; pf = 'Package Control.sublime-package'; ipp = sublime.installed_packages_path(); urllib.request.install_opener( urllib.request.build_opener( urllib.request.ProxyHandler()) ); by = urllib.request.urlopen( 'http://packagecontrol.io/' + pf.replace(' ', '%20')).read(); dh = hashlib.sha256(by).hexdigest(); print('Error validating download (got %s instead of %s), please try manual install' % (dh, h)) if dh != h else open(os.path.join( ipp, pf), 'wb' ).write(by)
之后的插件通过 PM 安装就可以了（Shift+Command+P）。我用到的有：
- Markdown Preview 有 github favor 的语法支持
- Pretty JSON
- Enhanced Clojure
我用到的插件比较少，毕竟我用的比较“轻”，只是进行一些临时性的编辑工作，主要的开发还是放在 Emacs 上。
数据库 GUI
# 安装命令
brew cask install sequel-pro # mysql
brew cask install robo-3t # mongodb
brew cask install rdm # redis
Mac 上自带的 Safari 比较轻量，虽然比较省电，但扩展性远不如 Chrome、Firefox，所以这两个是必须的。
brew cask install firefox
brew cask install google-chrome
# 现在的 Edge 完全兼容 Chrome，并且不用翻墙
brew cask install microsoft-edge
Chrome 默认会安装一个 Update 程序，在 ~/Library/Google/GoogleSoftwareUpdate，可以执行下面命令删除：
cd /Users/liujiacai/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle/Contents/Resources/GoogleSoftwareUpdateAgent.app/Contents/Resources
./ksinstall --nuke
对于程序员，如果上面两个浏览器不够用，可以安装相应的开发版：
brew cask install chromium
brew cask install google-chrome-canary # 金丝雀版
brew cask install firefoxdeveloperedition
必备插件
- Violentmonkey 里面的脚本通过 OneNote 同步
- Dark Reader
- SwitchyOmega
- Clearly
- 仿生学阅读 BionicReading
- JSON Formatter
- uBlock Origin
userscript
百度网盘选中一行的文件
var main = document.getElementById('layoutMain');
var dds = main.children[0].children[1].children[0].children[3].querySelectorAll('dd.g-clearfix');
// HTMLCollection isn't array, so hack it!
[].slice.call(dds[0].children).forEach((e,i) => console.log(e.children[2].click()));
截屏
brew cask install snip
Mac上的截图工具已经很好了，Cmd + Shift + 3/4就够用了，但是如果想在图片上写些文字，马赛克某部分，就不行了，推荐用 Snip，才 2M 大小，虽说是腾讯开发的，但是不流氓。可以设置快捷键，我设定的是Cmd + Shift + 6。
更重要的一点是，Snip 可以解决Retina下截屏2x问题（就是截出来的图超大），就光这个特点就足以让你使用snip。
马赛克
圈点洋名 Skitch，出自 Evernote，可以方便打马赛克，加标注。
brew cask install skitch
图片处理
Mac 版的 PS ImageMagick，常用命令：
brew install ImageMagick
# 下面两条命令都会把 left.png right.png 合并到 merged.png 里面
convert +append left.png right.png merged.png
montage -geometry 100% left.jpg right.jpg merged.jpg
# 批量格式转化，把 iOS 拍照默认的 HEIC 格式转为 jpg
find . -name '*HEIC' -exec echo convert {} {}.jpg \;
exif 元信息
# 查看 exif
mdls xxx.jpg
# 修改 exif
brew install exiftool
# 去掉拍摄日期
exiftool -createdate= -modifydate= a.jpg
exiftool -all a.jpg
录屏 gif
# 安装命令
brew cask install licecap
很多时候我们需要把自己的操作展示给别人看，比较好的做法是通过录屏软件将自己的操作保存成 gif 格式的图片。 开源免费的licecap 很好的解决了这个问题。
流程图制作工具
对于程序员来说，流程图应该是再亲切不过的了，一张图胜过千言万语。之前我都是用 Keynote 来画，但是实在是不好用，后来在知乎上发现了在线版的ProcessOn，大大减少了我画流程图的时间，上手也比较快。现在 ProcessOn 有了限制，只能保留 9 张流程图。我又找到了新的工具，draw.io，时序图、状态图统统不在话下。
国外很多项目的图是用纯文本画的，比较好用的在线工具是：asciiflow。
其次，Graphviz 提供了编程接口，也推荐使用，他也有网页版.
brew install Graphviz
视频播放器、截取
# 安装命令
brew cask install vlc
Mac下的自带的播放器QuickTime，功能实在是太弱了，支持的格式既少又难用，快进什么的貌似都没快捷键，只能手动点击进度条，试用了一段时间的Mplayer，发现效果也不好，会有视频卡顿的现象，最终选择了 VLC，一直用的还不错。 此外， 有网友补充道 mpv 才是程序员最佳播放器，大家也可以尝试下。
很多时候，我们只需要截取视频中的某一段视频，或者简单的进行格式转换，这时候就需要 ffmpeg 出马了。
# 安装命令
brew install ffmpeg
# 将 mov 格式的视频转为 mp4，ffmpeg 能根据文件后缀名自动识别
ffmpeg -i foo.mov foo.mp4
# 从第 6 秒开始，截取10s 视频，并且转为 mp4 格式
ffmpeg -t 10 -ss 00:00:06 -i foo.mov smaller.mp4
音乐频播放器
# 安装命令
brew cask install vox
官方的 iTunes 实在是不适应，喜欢简洁清爽的朋友可以试试 VOX
Calibre mobi 转化器
Calibre 电子书管理工具，功能非常全面，支持各种电子书格式的转换。
# 安装命令
brew cask install calibre
==> Installing Cask calibre
==> Moving App 'calibre.app' to '/Applications/calibre.app'.
==> Linking Binary 'calibre' to '/usr/local/bin/calibre'.
==> Linking Binary 'calibre-complete' to '/usr/local/bin/calibre-complete'.
==> Linking Binary 'calibre-customize' to '/usr/local/bin/calibre-customize'.
==> Linking Binary 'calibre-debug' to '/usr/local/bin/calibre-debug'.
==> Linking Binary 'calibre-parallel' to '/usr/local/bin/calibre-parallel'.
==> Linking Binary 'calibre-server' to '/usr/local/bin/calibre-server'.
==> Linking Binary 'calibre-smtp' to '/usr/local/bin/calibre-smtp'.
==> Linking Binary 'calibredb' to '/usr/local/bin/calibredb'.
==> Linking Binary 'ebook-convert' to '/usr/local/bin/ebook-convert'.
==> Linking Binary 'ebook-device' to '/usr/local/bin/ebook-device'.
==> Linking Binary 'ebook-edit' to '/usr/local/bin/ebook-edit'.
==> Linking Binary 'ebook-meta' to '/usr/local/bin/ebook-meta'.
==> Linking Binary 'ebook-polish' to '/usr/local/bin/ebook-polish'.
==> Linking Binary 'ebook-viewer' to '/usr/local/bin/ebook-viewer'.
==> Linking Binary 'fetch-ebook-metadata' to '/usr/local/bin/fetch-ebook-metadata'.
==> Linking Binary 'lrf2lrs' to '/usr/local/bin/lrf2lrs'.
==> Linking Binary 'lrfviewer' to '/usr/local/bin/lrfviewer'.
==> Linking Binary 'lrs2lrf' to '/usr/local/bin/lrs2lrf'.
==> Linking Binary 'markdown-calibre' to '/usr/local/bin/markdown-calibre'.
==> Linking Binary 'web2disk' to '/usr/local/bin/web2disk'.
k2pdfopt
- http://www.willus.com/k2pdfopt/download/
这个软件可以对一般普通的电子书针对移动设备优化，极大提高阅读体验，谁用谁知道！
PDF 切割
通过邮件发送 PDF 书籍到 kindle 时，有最大 50M 的限制，所以对于再大的书籍，就需要用数据线进行传输了，但是现在谁还有数据线呀，所以我选择的做法是对 PDF 进行切割！
sudo apt-get install pdftk
# 把第5页输出到一个新 PDF 文件中
pdftk ORIG_FILE.pdf cat 5 output NEW_FILE.pdf
# 范围为 1-5 页
pdftk ORIG_FILE.pdf cat 1-5 output NEW_FILE.pdf
pdftk ORIG_FILE.pdf cat 1 5 7 10-12 output NEW_FILE.pdf
- https://www.zhihu.com/question/37247669/answer/493037658
压缩软件
brew cask install the-unarchiver
使用 Windows 资源管理器创建的 zip 压缩包不保存文件名编码信息，在 Mac OS X 上使用默认解压缩工具可能会遇到中日韩文件名乱码。The Unarchiver 支持文件名编码检测，可最大程度解决文件名乱码问题。The Unarchiver 可以有效解决中文压缩文件的乱码问题。
下载软件 aria2
brew install aria2
aria2 是一个命令行下载软件，支持多种下载协议（HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink），允许多线程下载，占用资源少，下载速度非常快。
- https://aria2.github.io/
日历/Timer
Stopwatch/Timer
- http://joaomoreno.github.io/thyme/
brew cask install thyme
Menu Bar Calendar
- https://www.mowglii.com/itsycal/
brew cask install Itsycal
清 dns 缓存
sudo killall -HUP mDNSResponder
- https://support.apple.com/en-us/HT202516
锁屏
习惯 Windows 的用户，有一个很方便的锁屏快捷键 Win + L，但是 Mac 在升级到某个版本后，突然间取消了锁屏的快捷键，可以参考知乎上这个解决方法：
如果希望 sleep 再次唤醒时需要输入密码，可以修改一处设置：
修改 maxfiles
- https://superuser.com/questions/302754/increase-the-maximum-number-of-open-file-descriptors-in-snow-leopard/1171023#1171023
不能写 NTFS
用惯了 Windows 的大家都习惯用 NTFS 文件系统格式，但是很遗憾，这个文件系统是微软自己搞得，不是开放的，所以 Mac 是不支持的，如果你以前的 NTFS 格式的硬盘放到 Mac 上，会发现只能进行读操作，不能写入，这属于正常现象，不要惊慌。
解决的方法也很简单，把移动硬盘格式化成FAT32(单个文件大小不能超过4G)或FAText 格式都可以，Mac 自带的磁盘工具就可以进行格式转化，当然你需要先把移动硬盘上的数据拷贝出来。
Wi-Fi 时常中断
Mac 生于乔帮主之手时，为了凸显尊贵，接口与一般的电脑有很大不同。常见的网线没办法直接连接 Mac 电脑，需要单独购买一个以太网转接器，所以大部分同学都是使用无线连接，但 Mac 这里应该是有个 bug，而且是很久的 bug，我用 Mac 两年了，偶尔会遇到几次，网上解决的方法有如下几种：
- 修改网络位置，不是其默认的“自动”就好
- 修改路由器，把无线信道改为6或9
- 关闭蓝牙，Mac 中，同时打开蓝牙与 Wi-Fi 会冲突。详情
如果你的 Mac 也遇到了 Wi-Fi 问题，可以试试上面三个解决方法。
du 磁盘大小
du -sh some-directory
检查的是这个目录的物理大小，一个 1K 的文件，会因 文件碎片/sector/block size 等因素，在磁盘上占用的空间一般都要大于 1K。
- https://superuser.com/questions/382120/mac-os-x-not-reporting-directory-sizes-correctly
pkg 文件
pkgutil --expand mystubbornpackage.pkg path/to/expand
- https://apple.stackexchange.com/a/15665/103966
dpkg
dpkg is a package manager for Debian-based systems. It can install, remove, and build packages, but unlike other package management systems, it cannot automatically download and install packages or their dependencies.
sudo apt-get install build-essential
# list all packages installed on the system, from a terminal prompt type:
dpkg -l
# see all the files the package installed onto your system, do this:
dpkg-query -L <package_name>
# see the files a .deb file will install
dpkg-deb -c <package_name.deb>
GCC 8
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install gcc-8 g++-8
gcc-8 --version
# 默认使用 gcc-8
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 800 --slave /usr/bin/g++ g++ /usr/bin/g++-8
- https://askubuntu.com/a/1087116
- http://tuxamito.com/wiki/index.php/Installing_newer_GCC_versions_in_Ubuntu
sudo update-alternatives --remove-all gcc
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 90 --slave /usr/bin/g++ g++ /usr/bin/g++-8
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-6 80 --slave /usr/bin/g++ g++ /usr/bin/g++-6
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 70 --slave /usr/bin/g++ g++ /usr/bin/g++-7
sudo update-alternatives --config gcc
oom-killer log
- https://stackoverflow.com/questions/9199731/understanding-the-linux-oom-killers-logs
Server guide
- https://help.ubuntu.com/lts/serverguide/
apt-get
为了提高下载速度，需要配置国内源 /etc/apt/sources.list
- https://lug.ustc.edu.cn/wiki/mirrors/help/ubuntu
- http://mirrors.163.com/.help/ubuntu.html
- https://mirror.tuna.tsinghua.edu.cn/help/ubuntu/
- https://mirrors.aliyun.com/help/ubuntu
sudo sed -i 's/archive.ubuntu.com/mirrors.ustc.edu.cn/g' /etc/apt/sources.list
安装
sudo apt-get install supervisor
service supervisor restart
配置
supervisord.conf
[unix_http_server]
file=/var/run/supervisor/supervisor.sock ; (the path to the socket file)
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
user=admin ; (default is current user, required if root)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = supervisord.d/*.ini
app.ini
[program:prom]
command=./prometheus-2.9.2.linux-amd64/prometheus --config.file=prometheus.yml
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
numprocs=1 ; number of processes copies to start (def 1)
directory=/home/admin/prom
umask=022 ; umask for process (default None)
priority=999 ; the relative start priority (default 999)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=10 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=TERM ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=admin
stdout_logfile=/home/admin/prom/logs/stdout.log
stdout_logfile_maxbytes=50MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=5 ; # of stdout logfile backups (default 10)
stderr_logfile=/home/admin/prom/logs/stderr.log
stderr_logfile_maxbytes=50MB ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=5 ; # of stderr logfile backups (default 10)
environment=A=1,B=2 ; process environment additions (def no adds)
serverurl=AUTO ; override serverurl computation (childutils)
misc
Change kernel parameters at runtime
- https://unix.stackexchange.com/a/123050/101540
Mongodb
Docker 安装
- https://hub.docker.com/r/library/mongo/tags/
- https://hub.docker.com/r/percona/percona-server-mongodb
docker run -d -p 27017:27017 percona/percona-server-mongodb:32 --storageEngine=rocksdb
mongo --eval "printjson(db.serverStatus())"
命令
use jiacai
db.createCollection('_Conversation')
conv = db.getCollection('_Conversation')
conv.insertOne({
name: "tst conv",
c: "jiacai",
m: ["tom", "jerry"],
createdAt: Date(),
updatedAt: Date()
})
- https://docs.mongodb.com/manual/reference/method/db.createCollection/
设置最大内存以及丢弃策略
maxmemory <bytes>
maxmemory-policy volatile-lru
# redis-cli 查看当前配置。不设置或者设置为0，64位系统不限制内存，32位系统最多使用3GB内存。
config get maxmemory
config get maxmemory-policy
目前 redis 里面有 6 种删除策略：
| 策略 | 含义 |
|---|---|
volatile-lru | 根据LRU算法生成的过期时间来删除。 |
| allkeys-lru | 根据LRU算法删除任何key。 |
| volatile-random | 根据过期设置来随机删除key。 |
| allkeys->random | 无差别随机删。 |
| volatile-ttl | 根据最近过期时间来删除（辅以TTL） |
| noeviction | 谁也不删，直接在写操作时返回错误。 |
Kafka
- https://kafka.apache.org/quickstart
# 描述一个 topic
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic test
# 创建 topic
bin/kafka-topics.sh --zookeeper localhost:2181 --create --replication-factor 1 --partitions 3 --topic test
# 查看所有 topic
bin/kafka-topics.sh --list --zookeeper localhost:2181
# 删除 topic
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
# 发消息
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
# 收消息
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
在 0.9.0 之前的版本中，group 的 offset 存在 zk 中，之后的版本放在 broker 中。
# 查看所有 groupID
bin/kafka-consumer-groups.sh -zookeeper localhost:2181 -list # Old consumers
bin/kafka-consumer-groups.sh -bootstrap-server localhost:9092 -list # New consumers
# 除了 list 外，还有 --describe --reset-offsets --delete
- http://pavelmakhov.com/2017/03/useful-kafka-commands
- https://dzone.com/articles/kafka-architecture-log-compaction
配置
server
delete.topic.enable=true # 允许物理删除
log.dirs=/opt/kafka-logs
# The maximum size of the log before deleting it
log.retention.bytes
# The number of hours to keep a log file before deleting it
log.retention.hours
Topic-Level Configs
- https://kafka.apache.org/documentation/#topicconfigs
# 增加一个配置
bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity-name test --alter --add-config x=y
# 删除一个配置
bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity-name my_topic_name --alter --delete-config x
Kestrel
安装
- https://github.com/jiacai2050/docker-alpine-kestrel
文档
- https://github.com/twitter-archive/kestrel-client
- http://twitter-archive.github.io/kestrel/docs/guide.html
客户端
Ruby
sudo apt install -y libsasl2-dev # ubuntu
sudo gem install kestrel-client
require 'kestrel'
q = Kestrel::Client.new('localhost:22133')
q.set('a_queue', 'foo')
q.get('a_queue') # => 'foo'
q.pook('a_queue') # => 'foo'
Memcached
STATS
DUMP_STATS
STATUS
GET <queue-name>[options]
监控
- http://blog.scoutapp.com/articles/2012/08/15/kestrel-monitoring
Hbase
standalone 方式启动的 hbase 自带 zk，如果想与其他服务共用 zk，那么需要配置伪分布式。主要有以下配置：
- hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///opt/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
</configuration>
- hbase-env.sh
export HBASE_MANAGES_ZK=false
之后再 bin/start-hbase.sh 启动就可以了。
CDH 国内代理
- http://cloudera.proxy.ustclug.org/cdh5/cdh/5/
DDL
-- hive中的分隔符
CREATE TABLE employees (
name STRING,
salary FLOAT,
subordinates ARRAY<STRING>,
deductions MAP<STRING, FLOAT>,
address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT>
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001' COLLECTION ITEMS TERMINATED BY '\002' MAP KEYS TERMINATED BY '\003'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;
-- 上传内部表partition数据
LOAD DATA LOCAL INPATH '${env:HOME}/california-employees' INTO TABLE employees
PARTITION (country = 'US', state = 'CA');
-- 上传外部表partition数据
ALTER TABLE log_messages ADD PARTITION(year = 2012, month = 1, day = 2) LOCATION 'hdfs://master_server/data/log_messages/2012/01/02';
-- 更改现有partition位置
ALTER TABLE log_messages PARTITION(year = 2011, month = 12, day = 2) SET LOCATION 's3n://ourbucket/logs/2011/01/02';
ALTER TABLE temp_wx_bill DROP IF EXISTS PARTITION(par=201409);
-- 把string类型的s转为int类型
... cast(s AS INT) ...;
!connect jdbc:hive2://localhost:10000 scott tiger org.apache.hive.jdbc.HiveDriver
-- 给表加注释
ALTER TABLE xigua.t1 SET TBLPROPERTIES ('comment' = '你好呀');
wordcount
CREATE TABLE docs (line STRING);
LOAD DATA [local] INPATH 'docs' OVERWRITE INTO TABLE docs;
CREATE TABLE word_counts AS
SELECT word, count(1) AS count FROM
(SELECT explode(split(line, '\s')) AS word FROM docs) w
GROUP BY word
ORDER BY word;
自定义udf
create function unicode2str as 'com.youzan.hive.udf.Unicode2String';
CREATE FUNCTION unicode2str AS 'com.youzan.hive.udf.Unicode2String' USING JAR 'hdfs:///apps/hive/udf/yz-udf-unicode2str.jar';
select unicode2str(message) from dev.order_item_offline
where order_no='E20160711221336096050352';
hive server2 内存参数配置
hive-env.sh 里面添加
if [ "$SERVICE" = "hiveserver2" ]; then
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -server -XX:NewSize=1g -XX:MaxNewSize=1g -Xms5g -Xmx5g -XX:PermSize=1024m -XX:MaxPermSize=1024m -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=12345"
fi
debug
hive -hiveconf hive.root.logger=DEBUG,console
namenode 目录结构
dfs.namenode.name.dir 下的目录结构
./current
./current/edits_0000000000000000001-0000000000000000004
./current/edits_0000000000000000005-0000000000000000349
./current/edits_0000000000000000350-0000000000000000387
./current/edits_0000000000000000388-0000000000000000910
./current/edits_0000000000000000911-0000000000000001579
./current/edits_0000000000000001580-0000000000000001581
./current/edits_0000000000000001582-0000000000000001583
./current/edits_0000000000000001584-0000000000000001585
./current/edits_0000000000000001586-0000000000000001587
./current/edits_0000000000000001588-0000000000000001597
./current/edits_0000000000000001598-0000000000000001746
./current/edits_0000000000000001747-0000000000000001748
./current/edits_inprogress_0000000000000001749
./current/fsimage_0000000000000001746
./current/fsimage_0000000000000001746.md5
./current/fsimage_0000000000000001748
./current/fsimage_0000000000000001748.md5
./current/seen_txid
./current/VERSION
./in_use.lock
$ cat current/VERSION
#Sat Apr 02 17:40:43 CST 2016
namespaceID=13319364
clusterID=CID-d9e2c744-0641-4525-a618-3e75b452b9e5
cTime=0
storageType=NAME_NODE
blockpoolID=BP-744456632-127.0.0.1-1459590043064
layoutVersion=-60
datanode 目录结构
data/dfs/data/
├── current
│ ├── BP-1079595417-192.168.2.45-1412613236271
│ │ ├── current
│ │ │ ├── VERSION
│ │ │ ├── finalized
│ │ │ │ └── subdir0│ │ │ │ └── subdir1
│ │ │ │ ├── blk_1073741825
│ │ │ │ └── blk_1073741825_1001.meta
│ │ │ │── lazyPersist
│ │ │ └── rbw
│ │ ├── dncp_block_verification.log.curr
│ │ ├── dncp_block_verification.log.prev
│ │ └── tmp
│ └── VERSION
└── in_use.lock
- http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_hdfs_admin_tools/content/hdfs_metadata_datanodes.html