-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathatom.xml
More file actions
446 lines (264 loc) · 281 KB
/
atom.xml
File metadata and controls
446 lines (264 loc) · 281 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>NiuNiu's Note</title>
<link href="/atom.xml" rel="self"/>
<link href="http://liu-xin.me/"/>
<updated>2021-01-08T15:03:29.734Z</updated>
<id>http://liu-xin.me/</id>
<author>
<name>NiuNiu</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>动态规划和摩尔投票法</title>
<link href="http://liu-xin.me/2019/04/03/%E5%8A%A8%E6%80%81%E8%A7%84%E5%88%92%E5%92%8C%E6%91%A9%E5%B0%94%E6%8A%95%E7%A5%A8%E6%B3%95/"/>
<id>http://liu-xin.me/2019/04/03/动态规划和摩尔投票法/</id>
<published>2019-04-03T09:41:37.000Z</published>
<updated>2021-01-08T15:03:29.734Z</updated>
<content type="html"><![CDATA[<h3 id="动态规划"><a href="#动态规划" class="headerlink" title="动态规划"></a>动态规划</h3><p>维基百科对<a href="https://zh.wikipedia.org/wiki/%E5%8A%A8%E6%80%81%E8%A7%84%E5%88%92" target="_blank" rel="noopener">动态规划</a>（Dynamic programming，简称DP）的定义是一种在数学、管理科学、计算机科学、经济学和生物信息学中使用的，<strong>通过把原问题分解为相对简单的子问题的方式求解复杂问题的方法</strong>。<br><a id="more"></a></p><h4 id="斐波那契数列"><a href="#斐波那契数列" class="headerlink" title="斐波那契数列"></a>斐波那契数列</h4><p><a href="https://zh.wikipedia.org/wiki/%E6%96%90%E6%B3%A2%E9%82%A3%E5%A5%91%E6%95%B0%E5%88%97" target="_blank" rel="noopener">斐波那契数列</a>是一个典型的可以把原问题分解为相对简单的子问题的方式求解复杂问题的例子。下面是斐波那契数列的数学定义:</p><ul><li>F(0)=0，F(1)=1</li><li>F(2)=F(1)+F(0)=1+0=1</li><li>F(n)=F(n-1)+F(n-2) (n>=2)<br>根据这个数学定义，我们可以用递归的方式很轻松的实现这个算法。<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"time"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">fib</span><span class="params">(n <span class="keyword">int</span>)</span> <span class="title">int</span></span> {</span><br><span class="line"><span class="keyword">if</span> n <= <span class="number">1</span> {</span><br><span class="line"><span class="keyword">return</span> n</span><br><span class="line">}</span><br><span class="line"><span class="keyword">return</span> fib(n<span class="number">-1</span>) + fib(n<span class="number">-2</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">start := time.Now().Unix()</span><br><span class="line">fib(<span class="number">45</span>)</span><br><span class="line">end := time.Now().Unix()</span><br><span class="line">fmt.Println(end-start)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></li></ul><p>上面代码我们求的是F(45)，代码非常的简单，但发现计算时间达到了6秒左右，效率十分低下，下面是根据刚刚的代码画出的一个F(5)的树状图：<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/fib5.jpg" alt="fib5"><br>从图中可以看出F(3)计算了2次，F(2)计算了3次，F(1)计算了4次，发生了很多重复计算，这也是造成效率低下的原因，要优化的思路就是去除掉这些不必要的重复计算。现在我们将每个子问题的计算结果存储起来，当再次碰到同一个子问题时，就可以直接从之前存储的结果中取值，就不用再次计算了。比如第一次碰到计算F(2)时，可以用一个字典把F(2)的计算结果存储起来，当再次碰到计算F(2)时就可以直接从字典中取值，改造后的代码如下：<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"fmt"</span></span><br><span class="line"><span class="string">"time"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> m = <span class="keyword">map</span>[<span class="keyword">int</span>]<span class="keyword">int</span>{<span class="number">0</span>:<span class="number">0</span>, <span class="number">1</span>:<span class="number">1</span>}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">fib</span><span class="params">(n <span class="keyword">int</span>)</span> <span class="title">int</span></span> {</span><br><span class="line"><span class="keyword">if</span> v, ok :=m[n]; ok {</span><br><span class="line"><span class="keyword">return</span> v</span><br><span class="line">}</span><br><span class="line">m[n<span class="number">-1</span>],m[n<span class="number">-2</span>] =fib(n<span class="number">-1</span>),fib(n<span class="number">-2</span>)</span><br><span class="line"><span class="keyword">return</span> m[n<span class="number">-1</span>]+m[n<span class="number">-2</span>]</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">start := time.Now().UnixNano()</span><br><span class="line">fib(<span class="number">45</span>)</span><br><span class="line">end := time.Now().UnixNano()</span><br><span class="line">fmt.Println(end-start)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>经过改造后再计算F(45)不到1秒。<strong>一旦某个给定子问题的解已经算出，则将其记忆化存储，以便下次需要同一个子问题解之时直接查表</strong>这也是动态规划的重要内容。</p><p>所以动态规划两个最主要的点是：</p><ul><li>将一个复杂的问题分解为若干多个子问题。</li><li>将每个子问题的结果存储起来，使每个子问题只解决一次。</li></ul><h4 id="House-Robber"><a href="#House-Robber" class="headerlink" title="House Robber"></a>House Robber</h4><p>下面是用动态规划的方法来解决 LeetCode 上一道名为 <a href="https://leetcode-cn.com/problems/house-robber/" target="_blank" rel="noopener">House Robber</a> 的题目：</p><blockquote><p>你是一个专业的小偷，计划偷窃沿街的房屋。每间房内都藏有一定的现金，影响你偷窃的唯一制约因素就是相邻的房屋装有相互连通的防盗系统，如果两间相邻的房屋在同一晚上被小偷闯入，系统会自动报警。<br>给定一个代表每个房屋存放金额的非负整数数组，计算你在不触动警报装置的情况下，能够偷窃到的最高金额。<br>假设现在各个房屋存放的金额分别为2、7、9、3、1，求最大能偷窃到的金额。</p></blockquote><p>我们用 P(n) 表示为总共有 n 间房屋时能偷取到的最大金额，用 r(n) 表示为第 n 间房屋中存放的金额。 当 n 为1时 P(1)=r(1)，n 为2时 P(2)=Max(r(1), r(2))。因为题目要求不能打劫相邻两间房，所以当有 n 间房时 P(n)=Max(P(n-2)+r(n), P(n-1))。用方程来表示就是：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">P(1)=r(1)</span><br><span class="line">P(2)=Max(r(1), r(2))</span><br><span class="line">P(n)=Max(P(n-2)+r(n), P(n-1))</span><br></pre></td></tr></table></figure></p><p>所以这个问题就被分解成了若干个子问题，下面是其代码实现：<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> <span class="string">"fmt"</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> m = <span class="keyword">map</span>[<span class="keyword">int</span>]<span class="keyword">int</span>{}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">rob</span><span class="params">(arr []<span class="keyword">int</span>)</span> <span class="title">int</span></span> {</span><br><span class="line">l := <span class="built_in">len</span>(arr)</span><br><span class="line"><span class="keyword">if</span> l <= <span class="number">0</span> {</span><br><span class="line"><span class="keyword">return</span> <span class="number">0</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> v,ok:=m[l<span class="number">-1</span>];ok{</span><br><span class="line"><span class="keyword">return</span> v</span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> l == <span class="number">1</span> {</span><br><span class="line">m[<span class="number">0</span>]=arr[<span class="number">0</span>]</span><br><span class="line"><span class="keyword">return</span> arr[<span class="number">0</span>]</span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> l == <span class="number">2</span> {</span><br><span class="line"><span class="keyword">if</span> arr[<span class="number">0</span>] >= arr[<span class="number">1</span>] {</span><br><span class="line">m[<span class="number">1</span>]=arr[<span class="number">0</span>]</span><br><span class="line"><span class="keyword">return</span> arr[<span class="number">0</span>]</span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line">m[<span class="number">1</span>]=arr[<span class="number">1</span>]</span><br><span class="line"><span class="keyword">return</span> arr[<span class="number">1</span>]</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line">a, b:= rob(arr[:l<span class="number">-2</span>])+arr[l<span class="number">-1</span>],rob(arr[:l<span class="number">-1</span>])</span><br><span class="line"><span class="keyword">if</span> a>=b{</span><br><span class="line">m[l<span class="number">-1</span>]=a</span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line">m[l<span class="number">-1</span>]=b</span><br><span class="line">}</span><br><span class="line"><span class="keyword">return</span> m[l<span class="number">-1</span>]</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">arr := []<span class="keyword">int</span>{<span class="number">2</span>,<span class="number">7</span>,<span class="number">9</span>,<span class="number">3</span>,<span class="number">1</span>}</span><br><span class="line">m[<span class="number">0</span>]=arr[<span class="number">0</span>]</span><br><span class="line">ret :=rob(arr)</span><br><span class="line">fmt.Println(ret)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>上面的代码就是我们根据方程无脑写出的算法就已经达到了偷窃最大金额的目的，但其实还是有一些优化空间的，我们要计算 P(n) 其实只需要记住之前的 P(n-2) 和 P(n-1)就够了，但我们其实将 P(1)、P(2)、…、P(n-2) 都记住了，带来了一些内存浪费，之所以会有这个问题是因为我们求解 P(n) 时会依次求解 P(n-1)、P(n-2)、…、P(1) 是一种<strong>自顶向下</strong>的求解方式，如果换成<strong>自底向上</strong>的求解方式可以写出如下代码：<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> <span class="string">"fmt"</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">rob</span><span class="params">(arr []<span class="keyword">int</span>)</span> <span class="title">int</span></span> {</span><br><span class="line">pre1, pre2 := <span class="number">0</span>, <span class="number">0</span></span><br><span class="line"><span class="keyword">for</span> _,v := <span class="keyword">range</span> arr {</span><br><span class="line"><span class="keyword">if</span> pre2+v >= pre1 {</span><br><span class="line">pre1,pre2 = pre2+v,pre1</span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line">pre1,pre2= pre1,pre1</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"><span class="keyword">return</span> pre1</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">arr := []<span class="keyword">int</span>{<span class="number">2</span>,<span class="number">7</span>,<span class="number">9</span>,<span class="number">3</span>,<span class="number">1</span>}</span><br><span class="line">ret :=rob(arr)</span><br><span class="line">fmt.Println(ret)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>上面的变量 pre1 和 pre2 分别表示 P(n-1) 和 P(n-2)，这样通过<strong>自底向上</strong>的方式求出了结果，比<strong>自顶向下</strong>的方式更节省内存。</p><p>所以动态规划需要记住的几个关键点是<strong>将复杂问题拆分成若干个子问题</strong>、<strong>记住子问题的结果</strong>、<strong>自顶向下</strong>、<strong>自底向上</strong>。</p><h3 id="摩尔投票法"><a href="#摩尔投票法" class="headerlink" title="摩尔投票法"></a>摩尔投票法</h3><p>假如有10个人参与投票，有的人投给A，有的人投给B，有的人投给C，当我们想要找出A、B、C谁得票最多时，我们可以将两个不同的投票作为一对进行删除，直到不能再删时然后再查看结果中还剩下的投票就是得票最多的那个。比如上述10个人的投票情况是[A,B,C,C,B,A,A,A,B,A]，下面是进行删除的过程：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[A,B,C,C,B,A,A,A,B,A]==>[C,C,B,A,A,A,B,A] //A,B为不同的投票所以可以</span><br><span class="line"> //作为一对进行删除</span><br><span class="line">[C,C,B,A,A,A,B,A]==>[C,A,A,A,B,A] //C,C为相同的投票所以不删除，然后</span><br><span class="line"> // 再依次向后查找发现C,B不同可以删除 </span><br><span class="line">[C,A,A,A,B,A]==>[A,A,B,A]</span><br><span class="line">[A,A,B,A]==>[A,A]</span><br></pre></td></tr></table></figure></p><p>通过不断的对不同的投票作为一对进行删除，投票结果中最后只剩下了[A,A]，所以A就是得票最多的。摩尔投票法的核心就是<strong>将序列中两个不同的元素进行抵消或删除，序列最后剩下一个元素或多个相同的元素，那么这个元素就是出现次数最多的元素</strong>。</p><h4 id="Majority-Element"><a href="#Majority-Element" class="headerlink" title="Majority Element"></a>Majority Element</h4><p>求众数就是摩尔投票法的一个典型运用场景，比如有下面这道算法题：</p><blockquote><p>给定一个大小为 n 的数组，找到其中的众数。众数是指在数组中出现次数大于 n/2 的元素。给定数组[2,2,1,1,1,2,2,4,5,2,3,2,2] 找出其众数。</p></blockquote><p>实现代码如下：<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> <span class="string">"fmt"</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">arr := []<span class="keyword">int</span>{<span class="number">2</span>,<span class="number">2</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">2</span>,<span class="number">2</span>,<span class="number">4</span>,<span class="number">5</span>,<span class="number">2</span>,<span class="number">3</span>,<span class="number">2</span>,<span class="number">2</span>}</span><br><span class="line">maj, count := arr[<span class="number">0</span>], <span class="number">1</span></span><br><span class="line"><span class="keyword">for</span> i:=<span class="number">1</span>;i<<span class="built_in">len</span>(arr);i++ {</span><br><span class="line"><span class="keyword">if</span> maj == arr[i] {</span><br><span class="line">count++</span><br><span class="line">} <span class="keyword">else</span> {</span><br><span class="line"><span class="keyword">if</span> count == <span class="number">0</span> {</span><br><span class="line">maj,count = arr[i],<span class="number">1</span></span><br><span class="line"><span class="keyword">continue</span></span><br><span class="line">}</span><br><span class="line">count--</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line">fmt.Println(maj)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>代码中先假定数组的第一个元素就是众数，并用一个变量 count 来记录这个众数出现的次数，当被迭代到的数与这个众数相同时 count 就加1，不同时就做抵消操作，即 count 减1，当 count 为0时，就将被迭代到的数设为新的众数并将 count 置1。</p><p>以上就是摩尔投票法的原理和应用。</p>]]></content>
<summary type="html">
<h3 id="动态规划"><a href="#动态规划" class="headerlink" title="动态规划"></a>动态规划</h3><p>维基百科对<a href="https://zh.wikipedia.org/wiki/%E5%8A%A8%E6%80%81%E8%A7%84%E5%88%92" target="_blank" rel="noopener">动态规划</a>（Dynamic programming，简称DP）的定义是一种在数学、管理科学、计算机科学、经济学和生物信息学中使用的，<strong>通过把原问题分解为相对简单的子问题的方式求解复杂问题的方法</strong>。<br>
</summary>
<category term="动态规划" scheme="http://liu-xin.me/tags/%E5%8A%A8%E6%80%81%E8%A7%84%E5%88%92/"/>
<category term="摩尔投票法" scheme="http://liu-xin.me/tags/%E6%91%A9%E5%B0%94%E6%8A%95%E7%A5%A8%E6%B3%95/"/>
</entry>
<entry>
<title>用 Consul 来做服务注册与服务发现</title>
<link href="http://liu-xin.me/2019/03/28/%E6%9D%A5Consul%E5%81%9A%E6%9C%8D%E5%8A%A1%E6%B3%A8%E5%86%8C%E4%B8%8E%E6%9C%8D%E5%8A%A1%E5%8F%91%E7%8E%B0/"/>
<id>http://liu-xin.me/2019/03/28/来Consul做服务注册与服务发现/</id>
<published>2019-03-28T07:14:38.000Z</published>
<updated>2021-01-08T15:03:07.019Z</updated>
<content type="html"><![CDATA[<p>服务注册与服务发现是在分布式服务架构中常常会涉及到的东西，业界常用的服务注册与服务发现工具有 <a href="https://zookeeper.apache.org/" target="_blank" rel="noopener">ZooKeeper</a>、<a href="https://coreos.com/etcd/" target="_blank" rel="noopener">etcd</a>、<a href="https://www.consul.io/" target="_blank" rel="noopener">Consul</a> 和 <a href="https://github.com/Netflix/eureka" target="_blank" rel="noopener">Eureka</a>。Consul 的主要功能有服务发现、健康检查、KV存储、安全服务沟通和多数据中心。Consul 与其他几个工具的区别可以在这里查看 <a href="https://www.consul.io/intro/vs/index.html" target="_blank" rel="noopener">Consul vs. Other Software</a>。<br><a id="more"></a></p><h3 id="为什么需要有服务注册与服务发现？"><a href="#为什么需要有服务注册与服务发现？" class="headerlink" title="为什么需要有服务注册与服务发现？"></a>为什么需要有服务注册与服务发现？</h3><p>假设在分布式系统中有两个服务 Service-A （下文以“S-A”代称）和 Service-B（下文以“S-B”代称），当 S-A 想调用 S-B 时，我们首先想到的时直接在 S-A 中请求 S-B 所在服务器的 IP 地址和监听的端口，这在服务规模很小的情况下是没有任何问题的，但是在服务规模很大每个服务不止部署一个实例的情况下是存在一些问题的，比如 S-B 部署了三个实例 S-B-1、S-B-2 和 S-B-3，这时候 S-A 想调用 S-B 该请求哪一个服务实例的 IP 呢？还是将3个服务实例的 IP 都写在 S-A 的代码里，每次调用 S-B 时选择其中一个 IP？这样做显得很不灵活，这时我们想到了 <code>Nginx</code> 刚好就能很好的解决这个问题，引入 <code>Nginx</code> 后现在的架构变成了如下图这样：<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/SA-N-SB.jpeg" alt=""><br>引入 Nginx 后就解决了 S-B 部署多个实例的问题，还做了 S-B 实例间的负载均衡。但现在的架构又面临了新的问题，分布式系统往往要保证高可用以及能做到动态伸缩，在引入 Nginx 的架构中，假如当 S-B-1 服务实例不可用时，Nginx 仍然会向 S-B-1 分配请求，这样服务就不可用，我们想要的是 S-B-1 挂掉后 Nginx 就不再向其分配请求，以及当我们新部署了 S-B-4 和 S-B-5 后，Nginx 也能将请求分配到 S-B-4 和 S-B-5，Nginx 要做到这样就要在每次有服务实例变动时去更新配置文件再重启 Nginx。这样看似乎用了 Nginx 也很不舒服以及还需要人工去观察哪些服务有没有挂掉，Nginx 要是有对服务的健康检查以及能够动态变更服务配置就是我们想要的工具，这就是服务注册与服务发现工具的用处。下面是引入服务注册与服务发现工具后的架构图：<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/service_discovery.jpg" alt=""></p><p>在这个架构中：</p><ul><li>首先 S-B 的实例启动后将自身的服务信息（主要是服务所在的 IP 地址和端口号）注册到注册工具中。不同注册工具服务的注册方式各不相同，后文会讲 Consul 的具体注册方式。</li><li>服务将服务信息注册到注册工具后，注册工具就可以对服务做健康检查，以此来确定哪些服务实例可用哪些不可用。</li><li>S-A 启动后就可以通过服务注册和服务发现工具获取到所有健康的 S-B 实例的 IP 和端口，并将这些信息放入自己的内存中，S-A 就可用通过这些信息来调用 S-B。</li><li>S-A 可以通过监听（Watch）注册工具来更新存入内存中的 S-B 的服务信息。比如 S-B-1 挂了，健康检查机制就会将其标为不可用，这样的信息变动就被 S-A 监听到了，S-A 就更新自己内存中 S-B-1 的服务信息。</li></ul><p>所以务注册与服务发现工具除了服务本身的服务注册和发现功能外至少还需要有健康检查和状态变更通知的功能。</p><h3 id="Consul"><a href="#Consul" class="headerlink" title="Consul"></a>Consul</h3><p>Consul 作为一种分布式服务工具，为了避免单点故障常常以集群的方式进行部署，在 Consul 集群的节点中分为 Server 和 Client 两种节点（所有的节点也被称为Agent），Server 节点保存数据，Client 节点负责健康检查及转发数据请求到 Server；Server 节点有一个 Leader 节点和多个 Follower 节点，Leader 节点会将数据同步到 Follower 节点，在 Leader 节点挂掉的时候会启动选举机制产生一个新的 Leader。</p><p>Client 节点很轻量且无状态，它以 RPC 的方式向 Server 节点做读写请求的转发，此外也可以直接向 Server 节点发送读写请求。下面是 Consul 的架构图：<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/consul-arch-420ce04a.png" alt=""><br>Consule 的安装和具体使用及其他详细内容可浏览<a href="https://www.consul.io/docs/index.html" target="_blank" rel="noopener">官方文档</a>。<br>下面是我用 Docker 的方式搭建了一个有3个 Server 节点和1个 Client 节点的 Consul 集群。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"># 这是第一个 Consul 容器，其启动后的 IP 为172.17.0.5</span><br><span class="line">docker run -d --name=c1 -p 8500:8500 -e CONSUL_BIND_INTERFACE=eth0 consul agent --server=true --bootstrap-expect=3 --client=0.0.0.0 -ui</span><br><span class="line"></span><br><span class="line">docker run -d --name=c2 -e CONSUL_BIND_INTERFACE=eth0 consul agent --server=true --client=0.0.0.0 --join 172.17.0.5</span><br><span class="line"></span><br><span class="line">docker run -d --name=c3 -e CONSUL_BIND_INTERFACE=eth0 consul agent --server=true --client=0.0.0.0 --join 172.17.0.5</span><br><span class="line"></span><br><span class="line">#下面是启动 Client 节点</span><br><span class="line">docker run -d --name=c4 -e CONSUL_BIND_INTERFACE=eth0 consul agent --server=false --client=0.0.0.0 --join 172.17.0.5</span><br></pre></td></tr></table></figure></p><p>启动容器时指定的环境变量 <code>CONSUL_BIND_INTERFACE</code> 其实就是相当于指定了 Consul 启动时 <code>--bind</code> 变量的参数，比如可以把启动 c1 容器的命令换成下面这样，也是一样的效果。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker run -d --name=c1 -p 8500:8500 -e consul agent --server=true --bootstrap-expect=3 --client=0.0.0.0 --bind='{{ GetInterfaceIP "eth0" }}' -ui</span><br></pre></td></tr></table></figure></p><p>操作 Consul 有 <a href="https://www.consul.io/docs/commands/index.html" target="_blank" rel="noopener">Commands</a> 和 <a href="https://www.consul.io/api/index.html" target="_blank" rel="noopener">HTTP API</a> 两种方式，进入任意一个容器执行 <code>consul members</code> 都可以有如下的输出，说明 Consul 集群就已经搭建成功了。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Node Address Status Type Build Protocol DC Segment</span><br><span class="line">2dcf0c824cf0 172.17.0.7:8301 alive server 1.4.4 2 dc1 <all></span><br><span class="line">64746cffa116 172.17.0.6:8301 alive server 1.4.4 2 dc1 <all></span><br><span class="line">77af7d94a8ca 172.17.0.5:8301 alive server 1.4.4 2 dc1 <all></span><br><span class="line">6c71148f0307 172.17.0.8:8301 alive client 1.4.4 2 dc1 <default></span><br></pre></td></tr></table></figure></p><h3 id="代码实践"><a href="#代码实践" class="headerlink" title="代码实践"></a>代码实践</h3><p>假设现在有一个用 Node.js 写的服务 node-server 需要通过 <a href="https://grpc.io/" target="_blank" rel="noopener">gRPC</a> 的方式调用一个用 Go 写的服务 go-server。<br>下面是用 Protobuf 定义的服务和数据类型文件 <code>hello.proto</code>。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">syntax = "proto3";</span><br><span class="line"></span><br><span class="line">package hello;</span><br><span class="line">option go_package = "hello";</span><br><span class="line"></span><br><span class="line">// The greeter service definition.</span><br><span class="line">service Greeter {</span><br><span class="line"> // Sends a greeting</span><br><span class="line"> rpc SayHello (stream HelloRequest) returns (stream HelloReply) {}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">// The request message containing the user's name.</span><br><span class="line">message HelloRequest {</span><br><span class="line"> string name = 1;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">// The response message containing the greetings</span><br><span class="line">message HelloReply {</span><br><span class="line"> string message = 1;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>用命令通过 Protobuf 的定义生成 Go 语言的代码：<code>protoc --go_out=plugins=grpc:./hello ./*.proto</code> 会在 hello 目录下得到 hello.pb.go 文件，然后在 hello.go 文件中实现我们定义的 RPC 服务。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// hello.go</span></span><br><span class="line"><span class="keyword">package</span> hello</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> <span class="string">"context"</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> GreeterServerImpl <span class="keyword">struct</span> {}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(g *GreeterServerImpl)</span> <span class="title">SayHello</span><span class="params">(c context.Context, h *HelloRequest)</span> <span class="params">(*HelloReply, error)</span></span> {</span><br><span class="line">result := &HelloReply{</span><br><span class="line">Message: <span class="string">"hello"</span> + h.GetName(),</span><br><span class="line">}</span><br><span class="line"><span class="keyword">return</span> result, <span class="literal">nil</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>下面是入口文件 <code>main.go</code>，主要是将我们定义的服务注册到 gRPC 中，并建了一个 <code>/ping</code> 接口用于之后 Consul 的健康检查。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"go-server/hello"</span></span><br><span class="line"><span class="string">"google.golang.org/grpc"</span></span><br><span class="line"><span class="string">"net"</span></span><br><span class="line"><span class="string">"net/http"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">lis1, _ := net.Listen(<span class="string">"tcp"</span>, <span class="string">":8888"</span>)</span><br><span class="line">lis2, _ := net.Listen(<span class="string">"tcp"</span>, <span class="string">":8889"</span>)</span><br><span class="line">grpcServer := grpc.NewServer()</span><br><span class="line">hello.RegisterGreeterServer(grpcServer, &hello.GreeterServerImpl{})</span><br><span class="line"><span class="keyword">go</span> grpcServer.Serve(lis1)</span><br><span class="line"><span class="keyword">go</span> grpcServer.Serve(lis2)</span><br><span class="line"></span><br><span class="line">http.HandleFunc(<span class="string">"/ping"</span>, <span class="function"><span class="keyword">func</span><span class="params">(res http.ResponseWriter, req *http.Request)</span></span>{</span><br><span class="line">res.Write([]<span class="keyword">byte</span>(<span class="string">"pong"</span>))</span><br><span class="line">})</span><br><span class="line">http.ListenAndServe(<span class="string">":8080"</span>, <span class="literal">nil</span>)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>至此 go-server 端的代码就全部编写完了，可以看出代码里面没有任何涉及到 Consul 的地方，用 Consul 做服务注册是可以做到对项目代码没有任何侵入性的。下面要做的是将 go-server 注册到 Consul 中。将服务注册到 Consul 可以通过直接调用 Consul 提供的 REST API 进行注册，还有一种对项目没有侵入的配置文件进行注册。Consul 服务配置文件的详细内容可以<a href="https://www.consul.io/docs/agent/services.html" target="_blank" rel="noopener">在此查看</a>。下面是我们通过配置文件进行服务注册的配置文件 <code>services.json</code>：<br><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"services"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"id"</span>: <span class="string">"hello1"</span>,</span><br><span class="line"> <span class="attr">"name"</span>: <span class="string">"hello"</span>,</span><br><span class="line"> <span class="attr">"tags"</span>: [</span><br><span class="line"> <span class="string">"primary"</span></span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"address"</span>: <span class="string">"172.17.0.9"</span>,</span><br><span class="line"> <span class="attr">"port"</span>: <span class="number">8888</span>,</span><br><span class="line"> <span class="attr">"checks"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"http"</span>: <span class="string">"http://172.17.0.9:8080/ping"</span>,</span><br><span class="line"> <span class="attr">"tls_skip_verify"</span>: <span class="literal">false</span>,</span><br><span class="line"> <span class="attr">"method"</span>: <span class="string">"GET"</span>,</span><br><span class="line"> <span class="attr">"interval"</span>: <span class="string">"10s"</span>,</span><br><span class="line"> <span class="attr">"timeout"</span>: <span class="string">"1s"</span></span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> },{</span><br><span class="line"> <span class="attr">"id"</span>: <span class="string">"hello2"</span>,</span><br><span class="line"> <span class="attr">"name"</span>: <span class="string">"hello"</span>,</span><br><span class="line"> <span class="attr">"tags"</span>: [</span><br><span class="line"> <span class="string">"second"</span></span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">"address"</span>: <span class="string">"172.17.0.9"</span>,</span><br><span class="line"> <span class="attr">"port"</span>: <span class="number">8889</span>,</span><br><span class="line"> <span class="attr">"checks"</span>: [</span><br><span class="line"> {</span><br><span class="line"> <span class="attr">"http"</span>: <span class="string">"http://172.17.0.9:8080/ping"</span>,</span><br><span class="line"> <span class="attr">"tls_skip_verify"</span>: <span class="literal">false</span>,</span><br><span class="line"> <span class="attr">"method"</span>: <span class="string">"GET"</span>,</span><br><span class="line"> <span class="attr">"interval"</span>: <span class="string">"10s"</span>,</span><br><span class="line"> <span class="attr">"timeout"</span>: <span class="string">"1s"</span></span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line"> ]</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>配置文件中的 <code>172.17.0.9</code> 代表的是 go-server 所在服务器的 IP 地址，<code>port</code> 就是服务监听的不同端口，<code>check</code> 部分定义的就是健康检查，Consul 会每隔 10秒钟请求一下 <code>/ping</code> 接口以此来判断服务是否健康。将这个配置文件复制到 c4 容器的 /consul/config 目录，然后执行<code>consul reload</code> 命令后配置文件中的 hello 服务就注册到 Consul 中去了。通过在宿主机执行<code>curl http://localhost:8500/v1/catalog/services\?pretty</code>就能看到我们注册的 hello 服务。<br>下面是 node-server 服务的代码:<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> grpc = <span class="built_in">require</span>(<span class="string">'grpc'</span>);</span><br><span class="line"><span class="keyword">const</span> axios = <span class="built_in">require</span>(<span class="string">'axios'</span>);</span><br><span class="line"><span class="keyword">const</span> protoLoader = <span class="built_in">require</span>(<span class="string">'@grpc/proto-loader'</span>);</span><br><span class="line"><span class="keyword">const</span> packageDefinition = protoLoader.loadSync(</span><br><span class="line"> <span class="string">'./hello.proto'</span>,</span><br><span class="line"> {</span><br><span class="line"> keepCase: <span class="literal">true</span>,</span><br><span class="line"> longs: <span class="built_in">String</span>,</span><br><span class="line"> enums: <span class="built_in">String</span>,</span><br><span class="line"> defaults: <span class="literal">true</span>,</span><br><span class="line"> oneofs: <span class="literal">true</span></span><br><span class="line"> });</span><br><span class="line"><span class="keyword">const</span> hello_proto = grpc.loadPackageDefinition(packageDefinition).hello;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">getRandNum</span> (<span class="params">min, max</span>) </span>{</span><br><span class="line"> min = <span class="built_in">Math</span>.ceil(min);</span><br><span class="line"> max = <span class="built_in">Math</span>.floor(max);</span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">Math</span>.floor(<span class="built_in">Math</span>.random() * (max - min + <span class="number">1</span>)) + min;</span><br><span class="line">};</span><br><span class="line"></span><br><span class="line"><span class="keyword">const</span> urls = []</span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">function</span> <span class="title">getUrl</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="keyword">if</span> (urls.length) <span class="keyword">return</span> urls[getRandNum(<span class="number">0</span>, urls.length<span class="number">-1</span>)];</span><br><span class="line"> <span class="keyword">const</span> { data } = <span class="keyword">await</span> axios.get(<span class="string">'http://172.17.0.5:8500/v1/health/service/hello'</span>);</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">const</span> item <span class="keyword">of</span> data) {</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">const</span> check <span class="keyword">of</span> item.Checks) {</span><br><span class="line"> <span class="keyword">if</span> (check.ServiceName === <span class="string">'hello'</span> && check.Status === <span class="string">'passing'</span>) {</span><br><span class="line"> urls.push(<span class="string">`<span class="subst">${item.Service.Address}</span>:<span class="subst">${item.Service.Port}</span>`</span>)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> urls[getRandNum(<span class="number">0</span>, urls.length - <span class="number">1</span>)];</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">function</span> <span class="title">main</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> url = <span class="keyword">await</span> getUrl();</span><br><span class="line"> <span class="keyword">const</span> client = <span class="keyword">new</span> hello_proto.Greeter(url, grpc.credentials.createInsecure());</span><br><span class="line"> </span><br><span class="line"> client.sayHello({<span class="attr">name</span>: <span class="string">'jack'</span>}, <span class="function"><span class="keyword">function</span> (<span class="params">err, response</span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'Greeting:'</span>, response.message);</span><br><span class="line"> }); </span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">main()</span><br></pre></td></tr></table></figure></p><p>代码中 172.17.0.5 地址为 c1 容器的 IP 地址，node-server 项目中直接通过 Consul 提供的 API 获得了 hello 服务的地址，拿到服务后我们需要过滤出健康的服务的地址，再随机从所有获得的地址中选择一个进行调用。代码中没有做对 Consul 的监听，监听的实现可以通过不断的轮询上面的那个 API 过滤出健康服务的地址去更新 <code>urls</code> 数组来做到。现在启动 node-server 就可以调用到 go-server 服务。<br>服务注册与发现给服务带来了动态伸缩的能力，也给架构增加了一定的复杂度。Consul 除了服务发现与注册外，在配置中心、分布式锁方面也有着很多的应用。</p>]]></content>
<summary type="html">
<p>服务注册与服务发现是在分布式服务架构中常常会涉及到的东西，业界常用的服务注册与服务发现工具有 <a href="https://zookeeper.apache.org/" target="_blank" rel="noopener">ZooKeeper</a>、<a href="https://coreos.com/etcd/" target="_blank" rel="noopener">etcd</a>、<a href="https://www.consul.io/" target="_blank" rel="noopener">Consul</a> 和 <a href="https://github.com/Netflix/eureka" target="_blank" rel="noopener">Eureka</a>。Consul 的主要功能有服务发现、健康检查、KV存储、安全服务沟通和多数据中心。Consul 与其他几个工具的区别可以在这里查看 <a href="https://www.consul.io/intro/vs/index.html" target="_blank" rel="noopener">Consul vs. Other Software</a>。<br>
</summary>
<category term="Consul" scheme="http://liu-xin.me/tags/Consul/"/>
<category term="gRPC" scheme="http://liu-xin.me/tags/gRPC/"/>
</entry>
<entry>
<title>用 RabbitMQ 的死信队列来做定时任务</title>
<link href="http://liu-xin.me/2018/10/05/%E7%94%A8-RabbitMQ-%E7%9A%84%E6%AD%BB%E4%BF%A1%E9%98%9F%E5%88%97%E6%9D%A5%E5%81%9A%E5%AE%9A%E6%97%B6%E4%BB%BB%E5%8A%A1/"/>
<id>http://liu-xin.me/2018/10/05/用-RabbitMQ-的死信队列来做定时任务/</id>
<published>2018-10-05T10:26:50.000Z</published>
<updated>2021-01-08T15:05:27.940Z</updated>
<content type="html"><![CDATA[<p>在开发中做定时任务是一个非常常见的业务场景，在代码层面 Node.js 可以用 setTimeout、setInerval 这种基础语法或用 <a href="https://www.npmjs.com/package/node-schedule" target="_blank" rel="noopener">node-schedule</a> 这些类似的库来达到部分目的，在第三方服务上可以用 Redis 的 Keyspace Notification 或 Linux 自身的 crontab 来做定时任务。RabbitMQ 作为一个消息中间件，使用其死信队列也可以达到做定时任务的目的。<a id="more"></a></p><p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/rabbitmq-1.png" alt=""></p><p>本文以 Node.js 作为演示语言，操作 RabbitMQ 使用的是 <a href="https://www.npmjs.com/package/amqplib" target="_blank" rel="noopener">amqplib</a>。</p><h3 id="死信队列"><a href="#死信队列" class="headerlink" title="死信队列"></a>死信队列</h3><p>RabbitMQ 中有一种交换器叫 DLX，全称为 Dead-Letter-Exchange，可以称之为死信交换器。当消息在一个队列中变成死信（dead message）之后，它会被重新发送到另外一个交换器中，这个交换器就是 DLX，绑定在 DLX 上的队列就称之为死信队列。<br>消息变成死信一般是以下几种情况：</p><ul><li>消息被拒绝，并且设置 requeue 参数为 false</li><li>消息过期</li><li>队列达到最大长度</li></ul><p>DLX 也是一个正常的交换器，和一般的交换器没有区别，它能在任何队列上被指定，实际上就是设置某个队列的属性。当这个队列存在死信时，RabbitMQ 就会自动地将这个消息重新发布到设置的 DLX 上去，进而被路由到另一个队列，即死信队列。要为某个队列添加 DLX，需要在创建这个队列的时候设置其<code>deadLetterExchange</code> 和 <code>deadLetterRoutingKey</code> 参数，<code>deadLetterRoutingKey</code> 参数可选，表示为 DLX 指定的路由键，如果没有特殊指定，则使用原队列的路由键。</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> amqp = <span class="built_in">require</span>(<span class="string">'amqplib'</span>);</span><br><span class="line"></span><br><span class="line"><span class="keyword">const</span> myNormalEx = <span class="string">'my_normal_exchange'</span>;</span><br><span class="line"><span class="keyword">const</span> myNormalQueue = <span class="string">'my_normal_queue'</span>;</span><br><span class="line"><span class="keyword">const</span> myDeadLetterEx = <span class="string">'my_dead_letter_exchange'</span>;</span><br><span class="line"><span class="keyword">const</span> myDeadLetterRoutingKey = <span class="string">'my_dead_letter_routing_key'</span>;</span><br><span class="line"><span class="keyword">let</span> connection, channel;</span><br><span class="line">amqp.connect(<span class="string">'amqp://localhost'</span>)</span><br><span class="line"> .then(<span class="function">(<span class="params">conn</span>) =></span> {</span><br><span class="line"> connection = conn;</span><br><span class="line"> <span class="keyword">return</span> conn.createChannel();</span><br><span class="line"> })</span><br><span class="line"> .then(<span class="function">(<span class="params">ch</span>) =></span> {</span><br><span class="line"> channel = ch;</span><br><span class="line"> ch.assertExchange(myNormalEx, <span class="string">'direct'</span>, { <span class="attr">durable</span>: <span class="literal">false</span> });</span><br><span class="line"> <span class="keyword">return</span> ch.assertQueue(myNormalQueue, {</span><br><span class="line"> exclusive: <span class="literal">false</span>,</span><br><span class="line"> deadLetterExchange: myDeadLetterEx,</span><br><span class="line"> deadLetterRoutingKey: myDeadLetterRoutingKey,</span><br><span class="line"> });</span><br><span class="line"> })</span><br><span class="line"> .then(<span class="function">(<span class="params">ok</span>) =></span> {</span><br><span class="line"> channel.bindQueue(ok.queue, myNormalEx);</span><br><span class="line"> channel.sendToQueue(ok.queue, Buffer.from(<span class="string">'hello'</span>));</span><br><span class="line"> setTimeout(<span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{ connection.close(); process.exit(<span class="number">0</span>) }, <span class="number">500</span>);</span><br><span class="line"> })</span><br><span class="line"> .catch(<span class="built_in">console</span>.error);</span><br></pre></td></tr></table></figure><p>上面的代码先声明了一个交换器 <code>myNormalEx</code>， 然后声明了一个队列 <code>myNormalQueue</code>，在声明该队列的时候通过设置其 <code>deadLetterExchange</code> 参数，为其添加了一个 DLX。所以当队列 <code>myNormalQueue</code> 中有消息成为死信后就会被发布到 <code>myDeadLetterEx</code> 中去。</p><h3 id="过期时间（TTL）"><a href="#过期时间（TTL）" class="headerlink" title="过期时间（TTL）"></a>过期时间（TTL）</h3><p>在 RabbbitMQ 中，可以对消息和队列设置过期时间。当通过队列属性设置过期时间时，队列中所有消息都有相同的过期时间。当对消息设置单独的过期时间时，每条消息的 TTL 可以不同。如果两种方法一起使用，则消息的 TTL 以两者之间较小的那个数值为准。消息在队列中的生存时间一旦超过设置的 TTL 值时，就会变成“死信”（Dead Message），消费者将无法再接收到该消息。</p><p>针对每条消息设置 TTL 是在发送消息的时候设置 <code>expiration</code> 参数，单位为毫秒。</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> amqp = <span class="built_in">require</span>(<span class="string">'amqplib'</span>);</span><br><span class="line"></span><br><span class="line"><span class="keyword">const</span> myNormalEx = <span class="string">'my_normal_exchange'</span>;</span><br><span class="line"><span class="keyword">const</span> myNormalQueue = <span class="string">'my_normal_queue'</span>;</span><br><span class="line"><span class="keyword">const</span> myDeadLetterEx = <span class="string">'my_dead_letter_exchange'</span>;</span><br><span class="line"><span class="keyword">const</span> myDeadLetterRoutingKey = <span class="string">'my_dead_letter_routing_key'</span>;</span><br><span class="line"><span class="keyword">let</span> connection, channel;</span><br><span class="line">amqp.connect(<span class="string">'amqp://localhost'</span>)</span><br><span class="line"> .then(<span class="function">(<span class="params">conn</span>) =></span> {</span><br><span class="line"> connection = conn;</span><br><span class="line"> <span class="keyword">return</span> conn.createChannel();</span><br><span class="line"> })</span><br><span class="line"> .then(<span class="function">(<span class="params">ch</span>) =></span> {</span><br><span class="line"> channel = ch;</span><br><span class="line"> ch.assertExchange(myNormalEx, <span class="string">'direct'</span>, { <span class="attr">durable</span>: <span class="literal">false</span> });</span><br><span class="line"> <span class="keyword">return</span> ch.assertQueue(myNormalQueue, {</span><br><span class="line"> exclusive: <span class="literal">false</span>,</span><br><span class="line"> deadLetterExchange: myDeadLetterEx,</span><br><span class="line"> deadLetterRoutingKey: myDeadLetterRoutingKey,</span><br><span class="line"> });</span><br><span class="line">})</span><br><span class="line"> .then(<span class="function">(<span class="params">ok</span>) =></span> {</span><br><span class="line"> channel.bindQueue(ok.queue, myNormalEx);</span><br><span class="line"> channel.sendToQueue(ok.queue, Buffer.from(<span class="string">'hello'</span>), { <span class="attr">expiration</span>: <span class="string">'4000'</span>});</span><br><span class="line"> setTimeout(<span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{ connection.close(); process.exit(<span class="number">0</span>) }, <span class="number">500</span>);</span><br><span class="line"> })</span><br><span class="line"> .catch(<span class="built_in">console</span>.error);</span><br></pre></td></tr></table></figure><p>上面的代码在向队列发送消息的时候，通过传递 <code>{ expiration: '4000'}</code> 将这条消息的过期时间设为了4秒，对消息设置4秒钟过期，这条消息并不一定就会在4秒钟后被丢弃或进入死信，只有当这条消息到达队首即将被消费时才会判断其是否过期，若未过期就会被消费者消费，若已过期就会被删除或者成为死信。</p><h3 id="定时任务"><a href="#定时任务" class="headerlink" title="定时任务"></a>定时任务</h3><p>因为队列中的消息过期后会成为死信，而死信又会被发布到该消息所在的队列的 DLX 上去，<strong>所以通过为消息设置过期时间，然后再消费该消息所在队列的 DLX 所绑定的队列，从而来达到定时处理一个任务的目的。</strong> 简单的讲就是当有一个队列 queue1，其 DLX 为 deadEx1，deadEx1 绑定了一个队列 deadQueue1，当队列 queue1 中有一条消息因过期成为死信时，就会被发布到 deadEx1 中去，通过消费队列 deadQueue1 中的消息，也就相当于消费的是 queue1 中的因过期产生的死信消息。</p><p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/dead-letter.png" alt=""></p><p>消费死信队列的代码如下：</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> amqp = <span class="built_in">require</span>(<span class="string">'amqplib'</span>);</span><br><span class="line"></span><br><span class="line"><span class="keyword">const</span> myDeadLetterEx = <span class="string">'my_dead_letter_exchange'</span>;</span><br><span class="line"><span class="keyword">const</span> myDeadLetterQueue = <span class="string">'my_dead_letter_queue'</span>;</span><br><span class="line"><span class="keyword">const</span> myDeadLetterRoutingKey = <span class="string">'my_dead_letter_routing_key'</span>;</span><br><span class="line"><span class="keyword">let</span> channel;</span><br><span class="line">amqp.connect(<span class="string">'amqp://localhost'</span>)</span><br><span class="line">.then(<span class="function">(<span class="params">conn</span>) =></span> {</span><br><span class="line"> <span class="keyword">return</span> conn.createChannel();</span><br><span class="line">})</span><br><span class="line">.then(<span class="function">(<span class="params">ch</span>) =></span> {</span><br><span class="line"> channel = ch;</span><br><span class="line"> ch.assertExchange(myDeadLetterEx, <span class="string">'direct'</span>, { <span class="attr">durable</span>: <span class="literal">false</span> });</span><br><span class="line"> <span class="keyword">return</span> ch.assertQueue(myDeadLetterQueue, { <span class="attr">exclusive</span>: <span class="literal">false</span> });</span><br><span class="line">})</span><br><span class="line">.then(<span class="function">(<span class="params">ok</span>) =></span> {</span><br><span class="line"> channel.bindQueue(ok.queue, myDeadLetterEx, myDeadLetterRoutingKey);</span><br><span class="line"> channel.consume(ok.queue, (msg) => {</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">" [x] %s: '%s'"</span>, msg.fields.routingKey, msg.content.toString());</span><br><span class="line"> }, { <span class="attr">noAck</span>: <span class="literal">true</span>})</span><br><span class="line">})</span><br><span class="line">.catch(<span class="built_in">console</span>.error);</span><br></pre></td></tr></table></figure><p>这里需要注意的是，如果声明的 <code>myDeadLetterEx</code> 是 direct 类型，那么在为其绑定队列的时候一定要指定 BindingKey，即这里的 <code>myDeadLetterRoutingKey</code>，如果不指定 Bindingkey，则需要将 <code>myDeadLetterEx</code> 声明为 fanout 类型。</p>]]></content>
<summary type="html">
<p>在开发中做定时任务是一个非常常见的业务场景，在代码层面 Node.js 可以用 setTimeout、setInerval 这种基础语法或用 <a href="https://www.npmjs.com/package/node-schedule" target="_blank" rel="noopener">node-schedule</a> 这些类似的库来达到部分目的，在第三方服务上可以用 Redis 的 Keyspace Notification 或 Linux 自身的 crontab 来做定时任务。RabbitMQ 作为一个消息中间件，使用其死信队列也可以达到做定时任务的目的。
</summary>
<category term="RabbitMQ" scheme="http://liu-xin.me/tags/RabbitMQ/"/>
</entry>
<entry>
<title>让Express支持async/await</title>
<link href="http://liu-xin.me/2017/10/07/%E8%AE%A9Express%E6%94%AF%E6%8C%81async-await/"/>
<id>http://liu-xin.me/2017/10/07/让Express支持async-await/</id>
<published>2017-10-07T08:25:20.000Z</published>
<updated>2021-01-08T15:04:39.140Z</updated>
<content type="html"><![CDATA[<p>随着 Node.js v8 的发布，Node.js 已原生支持 async/await 函数，Web 框架 Koa 也随之发布了 Koa 2 正式版，支持 async/await 中间件，为处理异步回调带来了极大的方便。<br><a id="more"></a><br>既然 Koa 2 已经支持 async/await 中间件了，为什么不直接用 Koa，而还要去改造 Express 让其支持 async/await 中间件呢？因为 Koa 2 正式版发布才不久，而很多老项目用的都还是 Express，不可能将其推倒用 Koa 重写，这样成本太高，但又想用到新语法带来的便利，那就只能对 Express 进行改造了，而且这种改造必须是对业务无侵入的，不然会带来很多的麻烦。</p><h3 id="直接使用-async-await"><a href="#直接使用-async-await" class="headerlink" title="直接使用 async/await"></a>直接使用 async/await</h3><p>让我们先来看下在 Express 中直接使用 async/await 函数的情况。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> express = <span class="built_in">require</span>(<span class="string">'express'</span>);</span><br><span class="line"><span class="keyword">const</span> app = express();</span><br><span class="line"><span class="keyword">const</span> { promisify } = <span class="built_in">require</span>(<span class="string">'util'</span>);</span><br><span class="line"><span class="keyword">const</span> { readFile } = <span class="built_in">require</span>(<span class="string">'fs'</span>);</span><br><span class="line"><span class="keyword">const</span> readFileAsync = promisify(readFile);</span><br><span class="line"> </span><br><span class="line">app.get(<span class="string">'/'</span>, <span class="keyword">async</span> <span class="function"><span class="keyword">function</span> (<span class="params">req, res, next</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> data = <span class="keyword">await</span> readFileAsync(<span class="string">'./package.json'</span>);</span><br><span class="line"> res.send(data.toString());</span><br><span class="line">});</span><br><span class="line"><span class="comment">// Error Handler</span></span><br><span class="line">app.use(<span class="function"><span class="keyword">function</span> (<span class="params">err, req, res, next</span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.error(<span class="string">'Error:'</span>, err);</span><br><span class="line"> res.status(<span class="number">500</span>).send(<span class="string">'Service Error'</span>);</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line">app.listen(<span class="number">3000</span>, <span class="string">'127.0.0.1'</span>, <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">`Server running at http://<span class="subst">${ <span class="keyword">this</span>.address().address }</span>:<span class="subst">${ <span class="keyword">this</span>.address().port }</span>/`</span>);</span><br><span class="line">});</span><br></pre></td></tr></table></figure></p><p>上面是没有对 Express 进行改造，直接使用 async/await 函数来处理请求，当请求<code>http://127.0.0.1:3000/</code>时，发现请求能正常请求，响应也能正常响应。这样似乎不对 Express 做任何改造也能直接使用 async/await 函数，但如果 async/await 函数里发生了错误能不能被我们的错误处理中间件处理呢？现在我们去读取一个不存在文件，例如将之前读取的<code>package.json</code>换成<code>age.json</code>。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">app.get(<span class="string">'/'</span>, <span class="keyword">async</span> <span class="function"><span class="keyword">function</span> (<span class="params">req, res, next</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> data = <span class="keyword">await</span> readFileAsync(<span class="string">'./age.json'</span>);</span><br><span class="line"> res.send(data.toString());</span><br><span class="line">});</span><br></pre></td></tr></table></figure></p><p>现在我们去请求<code>http://127.0.0.1:3000/</code>时，发现请求迟迟不能响应，最终会超时。而在终端报了如下的错误：<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/unhandlererror.png" alt="UnhandlerRejectionError"><br>发现错误并没有被错误处理中间件处理，而是抛出了一个<code>unhandledRejection</code>异常，现在如果我们用 try/catch 来手动捕获错误会是什么情况呢？<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">app.get(<span class="string">'/'</span>, <span class="keyword">async</span> <span class="function"><span class="keyword">function</span> (<span class="params">req, res, next</span>) </span>{</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="keyword">const</span> data = <span class="keyword">await</span> readFileAsync(<span class="string">'./age.json'</span>);</span><br><span class="line"> res.send(datas.toString());</span><br><span class="line"> } <span class="keyword">catch</span>(e) {</span><br><span class="line"> next(e);</span><br><span class="line"> }</span><br><span class="line">});</span><br></pre></td></tr></table></figure></p><p>发现请求被错误处理中间件处理了，说明我们手动显式的来捕获错误是可以的，但是如果在每个中间件或请求处理函数里面加一个 try/catch 也太不优雅了，对业务代码有一定的侵入性，代码也显得难看。所以通过直接使用 async/await 函数的实验，我们发现对 Express 改造的方向就是能够接收 async/await 函数里面抛出的错误，又对业务代码没有侵入性。</p><h3 id="改造-Express"><a href="#改造-Express" class="headerlink" title="改造 Express"></a>改造 Express</h3><p>在 Express 中有两种方式来处理路由和中间件，一种是通过 Express 创建的 app，直接在 app 上添加中间件和处理路由，像下面这样：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> express = <span class="built_in">require</span>(<span class="string">'express'</span>);</span><br><span class="line"><span class="keyword">const</span> app = express();</span><br><span class="line"> </span><br><span class="line">app.use(<span class="function"><span class="keyword">function</span> (<span class="params">req, res, next</span>) </span>{</span><br><span class="line"> next();</span><br><span class="line">});</span><br><span class="line">app.get(<span class="string">'/'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">req, res, next</span>) </span>{</span><br><span class="line"> res.send(<span class="string">'hello, world'</span>);</span><br><span class="line">});</span><br><span class="line">app.post(<span class="string">'/'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">req, res, next</span>) </span>{</span><br><span class="line"> res.send(<span class="string">'hello, world'</span>);</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line">app.listen(<span class="number">3000</span>, <span class="string">'127.0.0.1'</span>, <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">`Server running at http://<span class="subst">${ <span class="keyword">this</span>.address().address }</span>:<span class="subst">${ <span class="keyword">this</span>.address().port }</span>/`</span>);</span><br><span class="line">});</span><br></pre></td></tr></table></figure></p><p>另外一种是通过 Express 的 Router 创建的路由实例，直接在路由实例上添加中间件和处理路由，像下面这样：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> express = <span class="built_in">require</span>(<span class="string">'express'</span>);</span><br><span class="line"><span class="keyword">const</span> app = express();</span><br><span class="line"><span class="keyword">const</span> router = <span class="keyword">new</span> express.Router();</span><br><span class="line">app.use(router);</span><br><span class="line"> </span><br><span class="line">router.get(<span class="string">'/'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">req, res, next</span>) </span>{</span><br><span class="line"> res.send(<span class="string">'hello, world'</span>);</span><br><span class="line">});</span><br><span class="line">router.post(<span class="string">'/'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">req, res, next</span>) </span>{</span><br><span class="line"> res.send(<span class="string">'hello, world'</span>);</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line">app.listen(<span class="number">3000</span>, <span class="string">'127.0.0.1'</span>, <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">`Server running at http://<span class="subst">${ <span class="keyword">this</span>.address().address }</span>:<span class="subst">${ <span class="keyword">this</span>.address().port }</span>/`</span>);</span><br><span class="line">});</span><br></pre></td></tr></table></figure></p><p>这两种方法可以混合起来用，现在我们思考一下怎样才能让一个形如<code>app.get('/', async function(req, res, next){})</code>的函数，让里面的 async 函数抛出的错误能被统一处理呢？要让错误被统一的处理当然要调用 <code>next(err)</code> 来让错误被传递到错误处理中间件，又由于 async 函数返回的是 Promise，所以肯定是形如这样的<code>asyncFn().then().catch(function(err){ next(err) })</code>，所以按这样改造一下就有如下的代码：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">app.get = <span class="function"><span class="keyword">function</span> (<span class="params">...data</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> params = [];</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> item <span class="keyword">of</span> data) {</span><br><span class="line"> <span class="keyword">if</span> (<span class="built_in">Object</span>.prototype.toString.call(item) !== <span class="string">'[object AsyncFunction]'</span>) {</span><br><span class="line"> params.push(item);</span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">const</span> handle = <span class="function"><span class="keyword">function</span> (<span class="params">...data</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> [ req, res, next ] = data;</span><br><span class="line"> item(req, res, next).then(next).catch(next);</span><br><span class="line"> };</span><br><span class="line"> params.push(handle);</span><br><span class="line"> }</span><br><span class="line"> app.get(...params)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>上面的这段代码中，我们判断<code>app.get()</code>这个函数的参数中，若有 async 函数，就采用<code>item(req, res, next).then(next).catch(next);</code>来处理，这样就能捕获函数内抛出的错误，并传到错误处理中间件里面去。但是这段代码有一个明显的错误就是最后调用 app.get()，这样就递归了，破坏了 app.get 的功能，也根本处理不了请求，因此还需要继续改造。<br>我们之前说 Express 两种处理路由和中间件的方式可以混用，那么我们就混用这两种方式来避免递归，代码如下：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> express = <span class="built_in">require</span>(<span class="string">'express'</span>);</span><br><span class="line"><span class="keyword">const</span> app = express();</span><br><span class="line"><span class="keyword">const</span> router = <span class="keyword">new</span> express.Router();</span><br><span class="line">app.use(router);</span><br><span class="line"> </span><br><span class="line">app.get = <span class="function"><span class="keyword">function</span> (<span class="params">...data</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> params = [];</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> item <span class="keyword">of</span> data) {</span><br><span class="line"> <span class="keyword">if</span> (<span class="built_in">Object</span>.prototype.toString.call(item) !== <span class="string">'[object AsyncFunction]'</span>) {</span><br><span class="line"> params.push(item);</span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">const</span> handle = <span class="function"><span class="keyword">function</span> (<span class="params">...data</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> [ req, res, next ] = data;</span><br><span class="line"> item(req, res, next).then(next).catch(next);</span><br><span class="line"> };</span><br><span class="line"> params.push(handle);</span><br><span class="line"> }</span><br><span class="line"> router.get(...params)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>像上面这样改造之后似乎一切都能正常工作了，能正常处理请求了。但通过查看 Express 的源码，发现这样破坏了 app.get() 这个方法，因为 app.get() 不仅能用来处理路由，而且还能用来获取应用的配置，在 Express 中对应的源码如下：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">methods.forEach(<span class="function"><span class="keyword">function</span>(<span class="params">method</span>)</span>{</span><br><span class="line"> app[method] = <span class="function"><span class="keyword">function</span>(<span class="params">path</span>)</span>{</span><br><span class="line"> <span class="keyword">if</span> (method === <span class="string">'get'</span> && <span class="built_in">arguments</span>.length === <span class="number">1</span>) {</span><br><span class="line"> <span class="comment">// app.get(setting)</span></span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">this</span>.set(path);</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">this</span>.lazyrouter();</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">var</span> route = <span class="keyword">this</span>._router.route(path);</span><br><span class="line"> route[method].apply(route, slice.call(<span class="built_in">arguments</span>, <span class="number">1</span>));</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">this</span>;</span><br><span class="line"> };</span><br><span class="line">});</span><br></pre></td></tr></table></figure></p><p>所以在改造时，我们也需要对 app.get 做特殊处理。在实际的应用中我们不仅有 get 请求，还有 post、put 和 delete 等请求，所以我们最终改造的代码如下：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> { promisify } = <span class="built_in">require</span>(<span class="string">'util'</span>);</span><br><span class="line"><span class="keyword">const</span> { readFile } = <span class="built_in">require</span>(<span class="string">'fs'</span>);</span><br><span class="line"><span class="keyword">const</span> readFileAsync = promisify(readFile);</span><br><span class="line"><span class="keyword">const</span> express = <span class="built_in">require</span>(<span class="string">'express'</span>);</span><br><span class="line"><span class="keyword">const</span> app = express();</span><br><span class="line"><span class="keyword">const</span> router = <span class="keyword">new</span> express.Router();</span><br><span class="line"><span class="keyword">const</span> methods = [ <span class="string">'get'</span>, <span class="string">'post'</span>, <span class="string">'put'</span>, <span class="string">'delete'</span> ];</span><br><span class="line">app.use(router);</span><br><span class="line"> </span><br><span class="line"><span class="keyword">for</span> (<span class="keyword">let</span> method <span class="keyword">of</span> methods) {</span><br><span class="line"> app[method] = <span class="function"><span class="keyword">function</span> (<span class="params">...data</span>) </span>{</span><br><span class="line"> <span class="keyword">if</span> (method === <span class="string">'get'</span> && data.length === <span class="number">1</span>) <span class="keyword">return</span> app.set(data[<span class="number">0</span>]);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">const</span> params = [];</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> item <span class="keyword">of</span> data) {</span><br><span class="line"> <span class="keyword">if</span> (<span class="built_in">Object</span>.prototype.toString.call(item) !== <span class="string">'[object AsyncFunction]'</span>) {</span><br><span class="line"> params.push(item);</span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">const</span> handle = <span class="function"><span class="keyword">function</span> (<span class="params">...data</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> [ req, res, next ] = data;</span><br><span class="line"> item(req, res, next).then(next).catch(next);</span><br><span class="line"> };</span><br><span class="line"> params.push(handle);</span><br><span class="line"> }</span><br><span class="line"> router[method](...params);</span><br><span class="line"> };</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">app.get(<span class="string">'/'</span>, <span class="keyword">async</span> <span class="function"><span class="keyword">function</span> (<span class="params">req, res, next</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> data = <span class="keyword">await</span> readFileAsync(<span class="string">'./package.json'</span>);</span><br><span class="line"> res.send(data.toString());</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line">app.post(<span class="string">'/'</span>, <span class="keyword">async</span> <span class="function"><span class="keyword">function</span> (<span class="params">req, res, next</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> data = <span class="keyword">await</span> readFileAsync(<span class="string">'./age.json'</span>);</span><br><span class="line"> res.send(data.toString());</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line">router.use(<span class="function"><span class="keyword">function</span> (<span class="params">err, req, res, next</span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.error(<span class="string">'Error:'</span>, err);</span><br><span class="line"> res.status(<span class="number">500</span>).send(<span class="string">'Service Error'</span>);</span><br><span class="line">}); </span><br><span class="line"> </span><br><span class="line">app.listen(<span class="number">3000</span>, <span class="string">'127.0.0.1'</span>, <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">`Server running at http://<span class="subst">${ <span class="keyword">this</span>.address().address }</span>:<span class="subst">${ <span class="keyword">this</span>.address().port }</span>/`</span>);</span><br><span class="line">});</span><br></pre></td></tr></table></figure></p><p>现在就改造完了，我们只需要加一小段代码，就可以直接用 async function 作为 handler 处理请求，对业务也毫无侵入性，抛出的错误也能传递到错误处理中间件。</p>]]></content>
<summary type="html">
<p>随着 Node.js v8 的发布，Node.js 已原生支持 async/await 函数，Web 框架 Koa 也随之发布了 Koa 2 正式版，支持 async/await 中间件，为处理异步回调带来了极大的方便。<br>
</summary>
<category term="Node.js" scheme="http://liu-xin.me/tags/Node-js/"/>
<category term="Express" scheme="http://liu-xin.me/tags/Express/"/>
</entry>
<entry>
<title>让写入数据库的数据自动写入缓存</title>
<link href="http://liu-xin.me/2017/03/24/%E8%AE%A9%E5%86%99%E5%85%A5%E6%95%B0%E6%8D%AE%E5%BA%93%E7%9A%84%E6%95%B0%E6%8D%AE%E8%87%AA%E5%8A%A8%E5%86%99%E5%85%A5%E7%BC%93%E5%AD%98/"/>
<id>http://liu-xin.me/2017/03/24/让写入数据库的数据自动写入缓存/</id>
<published>2017-03-24T06:56:11.000Z</published>
<updated>2018-10-06T16:04:38.288Z</updated>
<content type="html"><![CDATA[<p>在项目开发中，为了减轻数据库的 I/O 压力，加快请求的响应速度，缓存是常用到的技术。<a href="https://redis.io/" target="_blank" rel="noopener">Redis</a> 和 <a href="https://memcached.org/" target="_blank" rel="noopener">Memcache</a> 是现在常用的两个用来做数据缓存的技术。<br><a id="more"></a><br>数据缓存一些常见的做法是，让数据写入到数据库以后通过一些自动化的脚本自动同步到缓存，或者在向数据库写数据后再手动向缓存写一次数据。这些做法不免都有些繁琐，且代码也不好维护。我在写 Node.js 项目的时候，发现<strong>利用 <a href="https://www.npmjs.com/package/mongoose" target="_blank" rel="noopener">Mongoose</a>（一个 MongoDB 的 ODM）和 <a href="https://www.npmjs.com/package/sequelize" target="_blank" rel="noopener">Sequelize</a>（一个 MySQL 的 ORM）的一些功能特性能够优雅的做到让写入到 MongoDB/MySQL 的数据自动写入到 Redis，并且在做查询操作的时候能够自动地优先从缓存中查找数据，若缓存中找不到才进入 DB 中查找，并将 DB 中找到的数据写入缓存。</strong></p><p>本文不讲解 Mongoose 和 Sequelize 的基本用法，这里只讲解如何做到上面所说的自动缓存。</p><blockquote><p>本文要用到的一些库为 <a href="https://www.npmjs.com/package/mongoose" target="_blank" rel="noopener">Mongoose</a>、<a href="https://www.npmjs.com/package/sequelize" target="_blank" rel="noopener">Sequelize</a>、<a href="https://www.npmjs.com/package/ioredis" target="_blank" rel="noopener">ioredis</a> 和 <a href="https://www.npmjs.com/package/lodash" target="_blank" rel="noopener">lodash</a>。Node.js 版本为 7.7.1。</p></blockquote><h3 id="在-MongoDB-中实现自动缓存"><a href="#在-MongoDB-中实现自动缓存" class="headerlink" title="在 MongoDB 中实现自动缓存"></a>在 MongoDB 中实现自动缓存</h3><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// redis.js</span></span><br><span class="line"><span class="keyword">const</span> Redis = <span class="built_in">require</span>(<span class="string">'ioredis'</span>);</span><br><span class="line"><span class="keyword">const</span> Config = <span class="built_in">require</span>(<span class="string">'../config'</span>);</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> redis = <span class="keyword">new</span> Redis(Config.redis);</span><br><span class="line"> </span><br><span class="line"><span class="built_in">module</span>.exports = redis;</span><br></pre></td></tr></table></figure><p>上面文件的代码主要用于连接 redis。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// mongodb.js</span></span><br><span class="line"><span class="keyword">const</span> mongoose = <span class="built_in">require</span>(<span class="string">'mongoose'</span>);</span><br><span class="line"> </span><br><span class="line">mongoose.Promise = global.Promise;</span><br><span class="line"><span class="keyword">const</span> demoDB = mongoose.createConnection(<span class="string">'mongodb://127.0.0.1/demo'</span>, {});</span><br><span class="line"> </span><br><span class="line"><span class="built_in">module</span>.exports = demoDB;</span><br></pre></td></tr></table></figure></p><p>上面是连接 mongodb 的代码。</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// mongoBase.js</span></span><br><span class="line"><span class="keyword">const</span> mongoose = <span class="built_in">require</span>(<span class="string">'mongoose'</span>);</span><br><span class="line"><span class="keyword">const</span> Schema = mongoose.Schema;</span><br><span class="line"><span class="keyword">const</span> redis = <span class="built_in">require</span>(<span class="string">'./redis'</span>);</span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">baseFind</span>(<span class="params">method, params, time</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> self = <span class="keyword">this</span>;</span><br><span class="line"> <span class="keyword">const</span> collectionName = <span class="keyword">this</span>.collection.name;</span><br><span class="line"> <span class="keyword">const</span> dbName = <span class="keyword">this</span>.db.name;</span><br><span class="line"> <span class="keyword">const</span> redisKey = [dbName, collectionName, <span class="built_in">JSON</span>.stringify(params)].join(<span class="string">':'</span>);</span><br><span class="line"> <span class="keyword">const</span> expireTime = time || <span class="number">3600</span>;</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> <span class="built_in">Promise</span>(<span class="function"><span class="keyword">function</span> (<span class="params">resolve, reject</span>) </span>{</span><br><span class="line"> redis.get(redisKey, <span class="function"><span class="keyword">function</span> (<span class="params">err, data</span>) </span>{</span><br><span class="line"> <span class="keyword">if</span> (err) <span class="keyword">return</span> reject(err);</span><br><span class="line"> <span class="keyword">if</span> (data) <span class="keyword">return</span> resolve(<span class="built_in">JSON</span>.parse(data));</span><br><span class="line"> </span><br><span class="line"> self[method](params).lean().exec(<span class="function"><span class="keyword">function</span> (<span class="params">err, data</span>) </span>{</span><br><span class="line"> <span class="keyword">if</span> (err) <span class="keyword">return</span> reject(err);</span><br><span class="line"> <span class="keyword">if</span> (<span class="built_in">Object</span>.keys(data).length === <span class="number">0</span>) <span class="keyword">return</span> resolve(data);</span><br><span class="line"> </span><br><span class="line"> redis.setex(redisKey, expireTime, <span class="built_in">JSON</span>.stringify(data));</span><br><span class="line"> resolve(data);</span><br><span class="line"> });</span><br><span class="line"> });</span><br><span class="line"> });</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> Methods = {</span><br><span class="line"> findCache(params, time) {</span><br><span class="line"> <span class="keyword">return</span> baseFind.call(<span class="keyword">this</span>, <span class="string">'find'</span>, params, time);</span><br><span class="line"> },</span><br><span class="line"> findOneCache(params, time) {</span><br><span class="line"> <span class="keyword">return</span> baseFind.call(<span class="keyword">this</span>, <span class="string">'findOne'</span>, params, time);</span><br><span class="line"> },</span><br><span class="line"> findByIdCache(params, time) {</span><br><span class="line"> <span class="keyword">return</span> baseFind.call(<span class="keyword">this</span>, <span class="string">'findById'</span>, params, time);</span><br><span class="line"> },</span><br><span class="line">};</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> BaseSchema = <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="keyword">this</span>.defaultOpts = {</span><br><span class="line"> };</span><br><span class="line">};</span><br><span class="line"> </span><br><span class="line">BaseSchema.prototype.extend = <span class="function"><span class="keyword">function</span> (<span class="params">schemaOpts</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> schema = <span class="keyword">this</span>.wrapMethods(<span class="keyword">new</span> Schema(schemaOpts, {</span><br><span class="line"> toObject: { <span class="attr">virtuals</span>: <span class="literal">true</span> },</span><br><span class="line"> toJSON: { <span class="attr">virtuals</span>: <span class="literal">true</span> },</span><br><span class="line"> }));</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> schema;</span><br><span class="line">};</span><br><span class="line"></span><br><span class="line">BaseSchema.prototype.wrapMethods = <span class="function"><span class="keyword">function</span> (<span class="params">schema</span>) </span>{</span><br><span class="line"> schema.post(<span class="string">'save'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">data</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> dbName = data.db.name;</span><br><span class="line"> <span class="keyword">const</span> collectionName = data.collection.name;</span><br><span class="line"> <span class="keyword">const</span> redisKey = [dbName, collectionName, <span class="built_in">JSON</span>.stringify(data._id)].join(<span class="string">':'</span>);</span><br><span class="line"> </span><br><span class="line"> redis.setex(redisKey, <span class="number">3600</span>, <span class="built_in">JSON</span>.stringify(<span class="keyword">this</span>));</span><br><span class="line"> });</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">Object</span>.keys(Methods).forEach(<span class="function"><span class="keyword">function</span> (<span class="params">method</span>) </span>{</span><br><span class="line"> schema.statics[method] = Methods[method];</span><br><span class="line"> });</span><br><span class="line"> <span class="keyword">return</span> schema;</span><br><span class="line">};</span><br><span class="line"> </span><br><span class="line"><span class="built_in">module</span>.exports = <span class="keyword">new</span> BaseSchema();</span><br></pre></td></tr></table></figure><p>上面的代码是在用 mongoose 建模的时候，所有的 schema 都会继承这个 BaseSchema。这个 BaseSchema 里面就为所有继承它的 schema 添加了一个模型在执行 save 方法后触发的文档中间件，这个中间件的作用就是在数据被写入 MongoDB 后再自动写入 redis。然后还为每个继承它的 schema 添加了三个静态方法，分别是 findByIdCache、findOneCache 和 findCache，它们分别是 findById、findOne 和 find 方法的扩展，只是不同之处在于用添加的三个方法进行查询时会根据传入的条件先从 redis 中查找数据，若查到就返回数据，若查不到就继续调用所对应的的原生方法进入 MongoDB 中查找，若在 MongoDB 中查到了，就把查到的数据写入 redis，供以后的查询使用。添加的这三个静态方法的调用方法和他们所对应的的原生方法一致，只是可以多传入一个时间，用来设置数据在缓存中的过期时间。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// userModel.js</span></span><br><span class="line"><span class="keyword">const</span> BaseSchema = <span class="built_in">require</span>(<span class="string">'./mongoBase'</span>);</span><br><span class="line"><span class="keyword">const</span> mongoDB = <span class="built_in">require</span>(<span class="string">'./mongodb.js'</span>);</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> userSchema = BaseSchema.extend({</span><br><span class="line"> name: <span class="built_in">String</span>,</span><br><span class="line"> age: <span class="built_in">Number</span>,</span><br><span class="line"> addr: <span class="built_in">String</span>,</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line"><span class="built_in">module</span>.exports = mongoDB.model(<span class="string">'User'</span>, userSchema, <span class="string">'user'</span>);</span><br></pre></td></tr></table></figure></p><p>这是为 user 集合建的一个模型，它就通过 BaseSchema.extend 方法继承了上面说到的中间件和静态方法。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// index.js</span></span><br><span class="line"><span class="keyword">const</span> UserModel = <span class="built_in">require</span>(<span class="string">'./userModl'</span>);</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> action = <span class="keyword">async</span> <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> user = <span class="keyword">await</span> UserModel.create({ <span class="attr">name</span>: <span class="string">'node'</span>, <span class="attr">age</span>: <span class="number">7</span>, <span class="attr">addr</span>: <span class="string">'nodejs.org'</span> });</span><br><span class="line"> <span class="keyword">const</span> data1 = <span class="keyword">await</span> UserModel.findByIdCache(user._id.toString());</span><br><span class="line"> <span class="keyword">const</span> data2 = <span class="keyword">await</span> UserModel.findOneCache({ <span class="attr">age</span>: <span class="number">7</span> });</span><br><span class="line"> <span class="keyword">const</span> data3 = <span class="keyword">await</span> UserModel.findCache({ <span class="attr">name</span>: <span class="string">'node'</span>, <span class="attr">age</span>: <span class="number">7</span> }, <span class="number">7200</span>);</span><br><span class="line"> <span class="keyword">return</span> [ data1, data2, data3];</span><br><span class="line">};</span><br><span class="line"> </span><br><span class="line">action().then(<span class="built_in">console</span>.log).catch(<span class="built_in">console</span>.error);</span><br></pre></td></tr></table></figure></p><p>上面的的代码就是向 User 集合中写了一条数据后，然后依次调用了我们自己添加的三个用于查询的静态方法。把 redis 的 monitor 打开，发现代码已经按我们预想的那样执行了。</p><p>总结，上面的方案主要通过 Mongoose 的中间件和静态方法达到了我们想要的功能。但添加的 findOneCache 和 findCache 方法很难达到较高的数据一致性，若要追求很强的数据一致性就用它们所对应的的 findOne 和 find。findByIdCache 能保证很好的数据一致性，但也仅限于修改数据时是查询出来再 save，若是直接 update 也做不到数据一致性。</p><h3 id="在-MySQL-中实现自动缓存"><a href="#在-MySQL-中实现自动缓存" class="headerlink" title="在 MySQL 中实现自动缓存"></a>在 MySQL 中实现自动缓存</h3><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// mysql.js</span></span><br><span class="line"><span class="keyword">const</span> Sequelize = <span class="built_in">require</span>(<span class="string">'sequelize'</span>);</span><br><span class="line"><span class="keyword">const</span> _ = <span class="built_in">require</span>(<span class="string">'lodash'</span>);</span><br><span class="line"><span class="keyword">const</span> redis = <span class="built_in">require</span>(<span class="string">'./redis'</span>);</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> setCache = <span class="function"><span class="keyword">function</span> (<span class="params">data</span>) </span>{</span><br><span class="line"> <span class="keyword">if</span> (_.isEmpty(data) || !data.id) <span class="keyword">return</span>;</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">const</span> dbName = data.$modelOptions.sequelize.config.database;</span><br><span class="line"> <span class="keyword">const</span> tableName = data.$modelOptions.tableName;</span><br><span class="line"> <span class="keyword">const</span> redisKey = [dbName, tableName, <span class="built_in">JSON</span>.stringify(data.id)].join(<span class="string">':'</span>)</span><br><span class="line"> redis.setex(redisKey, <span class="number">3600</span>, <span class="built_in">JSON</span>.stringify(data.toJSON()));</span><br><span class="line">};</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> sequelize = <span class="keyword">new</span> Sequelize(<span class="string">'demo'</span>, <span class="string">'root'</span>, <span class="string">''</span>, {</span><br><span class="line"> host: <span class="string">'localhost'</span>,</span><br><span class="line"> port: <span class="number">3306</span>,</span><br><span class="line"> hooks: {</span><br><span class="line"> afterUpdate(data) {</span><br><span class="line"> setCache(data);</span><br><span class="line"> },</span><br><span class="line"> afterCreate(data) {</span><br><span class="line"> setCache(data);</span><br><span class="line"> },</span><br><span class="line"> },</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line">sequelize</span><br><span class="line"> .authenticate()</span><br><span class="line"> .then(<span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">'Connection has been established successfully.'</span>);</span><br><span class="line"> })</span><br><span class="line"> .catch(<span class="function"><span class="keyword">function</span> (<span class="params">err</span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.error(<span class="string">'Unable to connect to the database:'</span>, err);</span><br><span class="line"> });</span><br><span class="line"> </span><br><span class="line"><span class="built_in">module</span>.exports = sequelize;</span><br></pre></td></tr></table></figure><p>上面的代码主要作用是连接 MySQL 并生成 sequelize 实例，在构建 sequelize 实例的时候添加了两个钩子方法 afterUpdate 和 afterCreate。afterUpdate 用于在模型实例更新后执行的函数，注意<strong>必须是模型实例更新</strong>才会触发此方法，如果是直接类似 Model.update 这种方式更新是不会触发这个钩子函数的，只能是一个已经存在的实例调用 save 方法的时候会触发这个钩子。afterCreate 是在模型实例创建后调用的钩子函数。这两个钩子的主要目的就是用来当一条数据被写入到 MySQL 后，再自动的写入到 redis，即实现自动缓存。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// mysqlBase.js</span></span><br><span class="line"><span class="keyword">const</span> _ = <span class="built_in">require</span>(<span class="string">'lodash'</span>);</span><br><span class="line"><span class="keyword">const</span> Sequelize = <span class="built_in">require</span>(<span class="string">'sequelize'</span>);</span><br><span class="line"><span class="keyword">const</span> redis = <span class="built_in">require</span>(<span class="string">'./redis'</span>);</span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">baseFind</span>(<span class="params">method, params, time</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> self = <span class="keyword">this</span>;</span><br><span class="line"> <span class="keyword">const</span> dbName = <span class="keyword">this</span>.sequelize.config.database;</span><br><span class="line"> <span class="keyword">const</span> tableName = <span class="keyword">this</span>.name;</span><br><span class="line"> <span class="keyword">const</span> redisKey = [dbName, tableName, <span class="built_in">JSON</span>.stringify(params)].join(<span class="string">':'</span>);</span><br><span class="line"> <span class="keyword">return</span> (<span class="keyword">async</span> <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> cacheData = <span class="keyword">await</span> redis.get(redisKey);</span><br><span class="line"> <span class="keyword">if</span> (!_.isEmpty(cacheData)) <span class="keyword">return</span> <span class="built_in">JSON</span>.parse(cacheData);</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">const</span> dbData = <span class="keyword">await</span> self[method](params);</span><br><span class="line"> <span class="keyword">if</span> (_.isEmpty(dbData)) <span class="keyword">return</span> {};</span><br><span class="line"> </span><br><span class="line"> redis.setex(redisKey, time || <span class="number">3600</span>, <span class="built_in">JSON</span>.stringify(dbData));</span><br><span class="line"> <span class="keyword">return</span> dbData;</span><br><span class="line"> })();</span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> Base = <span class="function"><span class="keyword">function</span> (<span class="params">sequelize</span>) </span>{</span><br><span class="line"> <span class="keyword">this</span>.sequelize = sequelize;</span><br><span class="line">};</span><br><span class="line"> </span><br><span class="line">Base.prototype.define = <span class="function"><span class="keyword">function</span> (<span class="params">model, attributes, options</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> self = <span class="keyword">this</span>;</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">this</span>.sequelize.define(model, _.assign({</span><br><span class="line"> id: {</span><br><span class="line"> type: Sequelize.UUID,</span><br><span class="line"> primaryKey: <span class="literal">true</span>,</span><br><span class="line"> defaultValue: Sequelize.UUIDV1,</span><br><span class="line"> },</span><br><span class="line"> }, attributes), _.defaultsDeep({</span><br><span class="line"> classMethods: {</span><br><span class="line"> findByIdCache(params, time) {</span><br><span class="line"> <span class="keyword">this</span>.sequelize = self.sequelize;</span><br><span class="line"> <span class="keyword">return</span> baseFind.call(<span class="keyword">this</span>, <span class="string">'findById'</span>, params, time);</span><br><span class="line"> },</span><br><span class="line"> findOneCache(params, time) {</span><br><span class="line"> <span class="keyword">this</span>.sequelize = self.sequelize;</span><br><span class="line"> <span class="keyword">return</span> baseFind.call(<span class="keyword">this</span>, <span class="string">'findOne'</span>, params, time);</span><br><span class="line"> },</span><br><span class="line"> findAllCache(params, time) {</span><br><span class="line"> <span class="keyword">this</span>.sequelize = self.sequelize;</span><br><span class="line"> <span class="keyword">return</span> baseFind.call(<span class="keyword">this</span>, <span class="string">'findAll'</span>, params, time);</span><br><span class="line"> },</span><br><span class="line"> },</span><br><span class="line"> }, options));</span><br><span class="line">};</span><br><span class="line"> </span><br><span class="line"><span class="built_in">module</span>.exports = Base;</span><br></pre></td></tr></table></figure></p><p>上面的代码同之前的 mongoBase 的作用大致一样。在 sequelize 建模的时候，所有的 schema 都会继承这个 Base。这个 Base 里面就为所有继承它的 schema 添加了三个静态方法，分别是 findByIdCache、findOneCahe 和 findAllCache，它们的作用和之前 mongoBase 中的那三个方法作用一样，只是为了和 sequelize 中原生的 findAll 保持一致，findCache 在这里变成了 findAllCache。在 sequelize 中为 schema 添加类方法（classMethods），即相当于在 mongoose 中为 schema 添加静态方法（statics)。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// mysqlUser.js</span></span><br><span class="line"><span class="keyword">const</span> Sequelize = <span class="built_in">require</span>(<span class="string">'sequelize'</span>);</span><br><span class="line"><span class="keyword">const</span> base = <span class="built_in">require</span>(<span class="string">'./mysqlBase.js'</span>);</span><br><span class="line"><span class="keyword">const</span> sequelize = <span class="built_in">require</span>(<span class="string">'./mysql.js'</span>);</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> Base = <span class="keyword">new</span> base(sequelize);</span><br><span class="line"><span class="built_in">module</span>.exports = Base.define(<span class="string">'user'</span>, {</span><br><span class="line"> name: Sequelize.STRING,</span><br><span class="line"> age: Sequelize.INTEGER,</span><br><span class="line"> addr: Sequelize.STRING,</span><br><span class="line">}, {</span><br><span class="line"> tableName: <span class="string">'user'</span>,</span><br><span class="line"> timestamps: <span class="literal">true</span>,</span><br><span class="line">});</span><br></pre></td></tr></table></figure></p><p>上面定义了一个 User schema，它就从 Base 中继承了findByIdCache、findOneCahe 和 findAllCache。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> UserModel = <span class="built_in">require</span>(<span class="string">'./mysqlUser'</span>);</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> action = <span class="keyword">async</span> <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="keyword">await</span> UserModel.sync({ <span class="attr">force</span>: <span class="literal">true</span> });</span><br><span class="line"> <span class="keyword">const</span> user = <span class="keyword">await</span> UserModel.create({ <span class="attr">name</span>: <span class="string">'node'</span>, <span class="attr">age</span>: <span class="number">7</span>, <span class="attr">addr</span>: <span class="string">'nodejs.org'</span> });</span><br><span class="line"> <span class="keyword">await</span> UserModel.findByIdCache(user.id);</span><br><span class="line"> <span class="keyword">await</span> UserModel.findOneCache({ <span class="attr">where</span>: { <span class="attr">age</span>: <span class="number">7</span> }});</span><br><span class="line"> <span class="keyword">await</span> UserModel.findAllCache({ <span class="attr">where</span>: { <span class="attr">name</span>: <span class="string">'node'</span>, <span class="attr">age</span>: <span class="number">7</span> }}, <span class="number">7200</span>);</span><br><span class="line"> <span class="keyword">return</span> <span class="string">'finish'</span>;</span><br><span class="line">};</span><br><span class="line"> </span><br><span class="line">action().then(<span class="built_in">console</span>.log).catch(<span class="built_in">console</span>.error);</span><br></pre></td></tr></table></figure></p><p>总结，通过 sequelize 实现的自动缓存方案和之前 mongoose 实现的一样，也会存在数据一致性问题，findByIdCache 较好，findOneCache 和 findAllCache 较差，当然这里很多细节考虑得不够完善，可根据业务合理调整。</p>]]></content>
<summary type="html">
<p>在项目开发中，为了减轻数据库的 I/O 压力，加快请求的响应速度，缓存是常用到的技术。<a href="https://redis.io/" target="_blank" rel="noopener">Redis</a> 和 <a href="https://memcached.org/" target="_blank" rel="noopener">Memcache</a> 是现在常用的两个用来做数据缓存的技术。<br>
</summary>
<category term="数据库" scheme="http://liu-xin.me/tags/%E6%95%B0%E6%8D%AE%E5%BA%93/"/>
<category term="MongoDB" scheme="http://liu-xin.me/tags/MongoDB/"/>
<category term="Node.js" scheme="http://liu-xin.me/tags/Node-js/"/>
<category term="MySQL" scheme="http://liu-xin.me/tags/MySQL/"/>
</entry>
<entry>
<title>由left-pad扯到JS中的位运算</title>
<link href="http://liu-xin.me/2017/01/07/%E7%94%B1left-pad%E6%89%AF%E5%88%B0JS%E4%B8%AD%E7%9A%84%E4%BD%8D%E8%BF%90%E7%AE%97/"/>
<id>http://liu-xin.me/2017/01/07/由left-pad扯到JS中的位运算/</id>
<published>2017-01-07T10:04:12.000Z</published>
<updated>2021-01-08T15:04:24.305Z</updated>
<content type="html"><![CDATA[<p>这个话题的由来是2016年3月份的时候 <a href="https://www.npmjs.com/" target="_blank" rel="noopener">NPM</a> 社区发生了‘left-pad’事件，不久后社区就有人发布了用来补救的，也是现在大家能用到的 <a href="https://www.npmjs.com/package/left-pad" target="_blank" rel="noopener">left-pad</a> 库。</p><a id="more"></a><p>最开始这个库的代码是这样的。</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">module</span>.exports = leftpad;</span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">leftpad</span> (<span class="params">str, len, ch</span>) </span>{</span><br><span class="line"> str = <span class="built_in">String</span>(str);</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">var</span> i = <span class="number">-1</span>;</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">if</span> (!ch && ch !== <span class="number">0</span>) ch = <span class="string">' '</span>;</span><br><span class="line"> </span><br><span class="line"> len = len - str.length;</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">while</span> (++i < len) {</span><br><span class="line"> str = ch + str;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> str;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>我第一次看到这段代码的时候，没看出什么毛病，觉得清晰明了。后来刷微博的时候<a href="http://weibo.com/p/1005051401880315/home?from=page_100505&mod=TAB#place" target="_blank" rel="noopener">@左耳朵耗子</a>老师指出了这段代码可以写得更有效率些，于是他就贴出了自己写的版本并给 left-pad 提了 PR，代码如下：</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">module</span>.exports = leftpad;</span><br><span class="line"> </span><br><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">leftpad</span> (<span class="params">str, len, ch</span>) </span>{</span><br><span class="line"> <span class="comment">//convert the `str` to String</span></span><br><span class="line"> str = str +<span class="string">''</span>;</span><br><span class="line"> </span><br><span class="line"> <span class="comment">//needn't to pad</span></span><br><span class="line"> len = len - str.length;</span><br><span class="line"> <span class="keyword">if</span> (len <= <span class="number">0</span>) <span class="keyword">return</span> str;</span><br><span class="line"> </span><br><span class="line"> <span class="comment">//convert the `ch` to String</span></span><br><span class="line"> <span class="keyword">if</span> (!ch && ch !== <span class="number">0</span>) ch = <span class="string">' '</span>;</span><br><span class="line"> ch = ch + <span class="string">''</span>;</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">var</span> pad = <span class="string">''</span>;</span><br><span class="line"> <span class="keyword">while</span> (<span class="literal">true</span>) {</span><br><span class="line"> <span class="keyword">if</span> (len & <span class="number">1</span>) pad += ch;</span><br><span class="line"> len >>= <span class="number">1</span>;</span><br><span class="line"> <span class="keyword">if</span> (len) ch += ch;</span><br><span class="line"> <span class="keyword">else</span> <span class="keyword">break</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> pad + str;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>我当时看到他的这段代码的里面的 <code>&</code>和<code>>></code>运算符的时候一下有点懵了，只知道这是位运算里面的‘按位与’和‘右移’运算，但是完全不知道为什么这样写就能提高效率。于是就想着去了解位运算的实质和使用场景。</p><p>在了解位运算之前，我们先必须了解一下什么是原码、反码和补码以及二进制与十进制的转换。</p><h3 id="原码、补码和反码"><a href="#原码、补码和反码" class="headerlink" title="原码、补码和反码"></a>原码、补码和反码</h3><h4 id="原码"><a href="#原码" class="headerlink" title="原码"></a>原码</h4><p>一个数在计算机中是以二进制的形式存在的，其中第一位存放符号, 正数为0, 负数为1。原码就是用第一位存放符号的二进制数值。例如2的原码为00000010，-2的原码为10000010。</p><h4 id="反码"><a href="#反码" class="headerlink" title="反码"></a>反码</h4><p>正数的反码是其本身。负数的反码是在其原码的基础上，符号位不变，其余各位取反，即0变1，1变0。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[+<span class="number">3</span>]=[<span class="number">00000011</span>]原=[<span class="number">00000011</span>]反</span><br><span class="line">[<span class="number">-3</span>]=[<span class="number">10000011</span>]原=[<span class="number">11111100</span>]反</span><br></pre></td></tr></table></figure></p><p>可见如果一个反码表示的是负数，并不能直观的看出它的数值，通常要将其转换成原码再计算。</p><h4 id="补码"><a href="#补码" class="headerlink" title="补码"></a>补码</h4><p>正数的补码是其本身。负数的补码是在其原码的基础上，符号位不变，其余各位取反，最后+1。（即负数的补码为在其反码的基础上+1）。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[+<span class="number">3</span>]=[<span class="number">00000011</span>]原=[<span class="number">00000011</span>]反=[<span class="number">00000011</span>]补</span><br><span class="line">[<span class="number">-3</span>]=[<span class="number">10000011</span>]原=[<span class="number">11111100</span>]反=[<span class="number">11111101</span>]补</span><br></pre></td></tr></table></figure></p><p>可见对于负数，补码的表示方式也是让人无法直观看出其数值的，通常也需要转换成原码再计算。</p><h3 id="二进制与十进制的转换"><a href="#二进制与十进制的转换" class="headerlink" title="二进制与十进制的转换"></a>二进制与十进制的转换</h3><p>二进制与十进制的区别在于数运算时是逢几进一位。二进制是逢2进一位，十进制也就是我们常用的0-9是逢10进一位</p><h4 id="正整数的十进制转二进制"><a href="#正整数的十进制转二进制" class="headerlink" title="正整数的十进制转二进制"></a>正整数的十进制转二进制</h4><p>正整数的十进制转二进制的方法为将一个十进制数除以2，得到的商再除以2，以此类推直到商等于1或0时为止，倒取除得的余数，即为转换所得的二进制数的结果。</p><p>例如把52换算成二进制数，计算过程如下图：<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/2to10.gif" alt="10to2"><br>52除以2得到的余数依次为：0、0、1、0、1、1，倒序排列，所以52对应的二进制数就是110100。</p><h4 id="负整数的十进制转二进制"><a href="#负整数的十进制转二进制" class="headerlink" title="负整数的十进制转二进制"></a>负整数的十进制转二进制</h4><p>负整数的十进制转二进制为将该负整数对应的正整数先转换成二进制，然后对其“取反”，再对取反后的结果+1。即负整数采用其二进制补码的形式存储。<br>至于负数为什么要用二进制补码的形式存储，可参考一篇阮一峰的文章《<a href="http://www.ruanyifeng.com/blog/2009/08/twos_complement.html" target="_blank" rel="noopener">关于2的补码</a>》。<br>例如 -52 的原码为 10110100，其反码为 11001011，其补码为 11001100。所以 -52 转换为二进制后为 11001100。</p><h4 id="十进制小数转二进制"><a href="#十进制小数转二进制" class="headerlink" title="十进制小数转二进制"></a>十进制小数转二进制</h4><p>十进制小数转二进制的方法为“乘2取整”，对十进制小数乘2得到的整数部分和小数部分，整数部分即是相应的二进制数码，再用2乘小数部分(之前乘后得到新的小数部分)，又得到整数和小数部分。<br>如此不断重复，直到小数部分为0或达到精度要求为止。第一次所得到为最高位，最后一次得到为最低位。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">如:0.25的二进制</span><br><span class="line">0.25*2=0.5 取整是0</span><br><span class="line">0.5*2=1.0 取整是1</span><br><span class="line">即0.25的二进制为 0.01 ( 第一次所得到为最高位,最后一次得到为最低位)</span><br><span class="line"> </span><br><span class="line">0.8125的二进制</span><br><span class="line">0.8125*2=1.625 取整是1</span><br><span class="line">0.625*2=1.25 取整是1</span><br><span class="line">0.25*2=0.5 取整是0</span><br><span class="line">0.5*2=1.0 取整是1</span><br><span class="line">即0.8125的二进制是0.1101</span><br></pre></td></tr></table></figure></p><h4 id="二进制转十进制"><a href="#二进制转十进制" class="headerlink" title="二进制转十进制"></a>二进制转十进制</h4><p>从最后一位开始算，依次列为第0、1、2…位，第n位的数（0或1）乘以2的n次方，将得到的结果相加就是得到的十进制数。<br>例如二进制为110的数，将其转为十进制的过程如下<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/2to10.png" alt="2to10"><br>个位数 0 与 2º 相乘：0 × 2º = 0<br>十位数 1 与 2¹ 相乘：1 × 2¹ = 2<br>百位数 1 与 2² 相乘：1 × 2² = 4<br>将得到的结果相加：0+2+4=6<br>所以二进制 110 转换为十进制后的数值为 6。</p><p>小数二进制用数值乘以2的负幂次然后相加。</p><h3 id="JavaScript-中的位运算"><a href="#JavaScript-中的位运算" class="headerlink" title="JavaScript 中的位运算"></a>JavaScript 中的位运算</h3><p>在 ECMAScript 中按位操作符会将其操作数转成<strong>补码形式的有符号32位整数。</strong>下面是<a href="http://www.ecma-international.org/ecma-262/6.0/#sec-binary-bitwise-operators-runtime-semantics-evaluation" target="_blank" rel="noopener">ECMAScript 规格</a>中对于位运算的执行过程的表述：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">The production A : A @ B, where @ is one of the bitwise operators in the productions above, is evaluated as follows:</span><br><span class="line">1. Let lref be the result of evaluating A.</span><br><span class="line">2. Let lval be GetValue(lref).</span><br><span class="line">3. ReturnIfAbrupt(lval).</span><br><span class="line">4. Let rref be the result of evaluating B.</span><br><span class="line">5. Let rval be GetValue(rref).</span><br><span class="line">6. ReturnIfAbrupt(rval).</span><br><span class="line">7. Let lnum be ToInt32(lval).</span><br><span class="line">8. ReturnIfAbrupt(lnum).</span><br><span class="line">9. Let rnum be ToInt32(rval).</span><br><span class="line">10. ReturnIfAbrupt(rnum).</span><br><span class="line">11. Return the result of applying the bitwise operator @ to lnum and rnum. The result is a signed 32 bit integer.</span><br></pre></td></tr></table></figure></p><p>需要注意的是第七步和第九步，根据 ES 的标准，超过32位的整数会被截断，而小数部分则会被直接舍弃。所以由此可以知道，在 JS 中，当位运算中有操作数大于或等于2³²时，就会出现意想不到的结果。</p><p>JavaScript 中的位运算有：<code>&（按位与）</code>、<code>|（按位或）</code>、<code>~（取反）</code>、<code>^（按位异或）</code>、<code><<（左移）</code>、<code>>>（有符号右移）</code>和<code>>>>（无符号右移）</code>。</p><h4 id="amp-按位与"><a href="#amp-按位与" class="headerlink" title="&按位与"></a>&按位与</h4><p>对每一个比特位执行与（AND）操作。只有 a 和 b 都是 1 时，a & b 才是 1。<br>例如：9(base 10) & 14(base 10) = 1001(base2) & 1110(base 2) = 1000(base 2) = 8(base 10)</p><p>因为当只有 a 和 b 都是 1 时，a&b才等于1，所以任一数值 x 与0（二进制的每一位都是0）按位与操作，其结果都为0。将任一数值 x 与 -1（二进制的每一位都是1）按位与操作，其结果都为 x。<br>利用 & 运算的特点，我们可以用以简单的判断奇偶数，公式：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">(n & <span class="number">1</span>) === <span class="number">0</span> <span class="comment">//true 为偶数，false 为奇数。</span></span><br></pre></td></tr></table></figure></p><p>因为 1 的二进制只有最后一位为1，其余位都是0，所以其判断奇偶的实质是判断二进制数最后一位是 0 还是 1。奇数的二进制最后一位是 1，偶数是0。</p><p>当然还可以利用 JS 在做位运算时会舍弃掉小数部分的特性来做向下取整的运算，因为当 x 为整数时有 <code>x&-1=x</code>，所以当 x 为小数时有 <code>x&-1===Math.floor(x)</code>。</p><h4 id="按位或"><a href="#按位或" class="headerlink" title="|按位或"></a>|按位或</h4><p>对每一个比特位执行或（OR）操作。如果 a 或 b 为 1，则 a | b 结果为 1。<br>例如：9(base 10) | 14(base 10) = 1001(base2) | 1110(base 2) = 1111(base 2) = 15(base 10)</p><p>因为只要 a 或 b 其中一个是 1 时，a|b就等于1，所以任一数值 x 与-1（二进制的每一位都是1）按位与操作，其结果都为-1。将任一数值 x 与 0（二进制的每一位都是0）按位与操作，其结果都为 x。</p><p>同样，按位或也可以做向下取整运算，因为当 x 为整数时有 <code>x|0=x</code>，所以当 x 为小数时有 <code>x|0===Math.floor(x)</code>。</p><h4 id="取反"><a href="#取反" class="headerlink" title="~取反"></a>~取反</h4><p>对每一个比特位执行非（NOT）操作。~a 结果为 a 的反转（即反码）。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">9 (base 10) = 00000000000000000000000000001001 (base 2)</span><br><span class="line"> --------------------------------</span><br><span class="line">~9 (base 10) = 11111111111111111111111111110110 (base 2) = -10 (base 10)</span><br></pre></td></tr></table></figure></p><p>负数的二进制转化为十进制的规则是，符号位不变，其他位取反后加 1。</p><p>对任一数值 x 进行按位非操作的结果为 -(x + 1)。~~x === x。</p><p>同样，取反也可以做向下取整运算，因为当 x 为整数时有 <code>~~x===x</code>，所以当 x 为小数时有 <code>~~x===Math.floor(x)</code>。</p><h4 id="按位异或"><a href="#按位异或" class="headerlink" title="^按位异或"></a>^按位异或</h4><p>对每一对比特位执行异或（XOR）操作。当 a 和 b 不相同时，a ^ b 的结果为 1。<br>例如：9(base 10) ^ 14(base 10) = 1001(base2) ^ 1110(base 2) = 0111(base 2) = 7(base 10)</p><p>将任一数值 x 与 0 进行异或操作，其结果为 x。将任一数值 x 与 -1 进行异或操作，其结果为 ~x，即 x^-1=~x。<br>同样，按位异或也可以做向下取整运算，因为当 x 为整数时有 <code>(x^0)===x</code>，所以当 x 为小数时有 <code>(x^0)===Math.floor(x)</code>。</p><h4 id="lt-lt-左移运算"><a href="#lt-lt-左移运算" class="headerlink" title="<<左移运算"></a><<左移运算</h4><p>它把数字中的所有数位向左移动指定的数量，向左被移出的位被丢弃，右侧用 0 补充。<br>例如，把数字 2（等于二进制中的 10）左移 5 位，结果为 64（等于二进制中的 1000000）：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> iOld = <span class="number">2</span>;<span class="comment">//等于二进制 10</span></span><br><span class="line"><span class="keyword">var</span> iNew = iOld << <span class="number">5</span>;<span class="comment">//等于二进制 1000000 十进制 64</span></span><br></pre></td></tr></table></figure></p><p>因为二进制10转换成十进制的过程为 1×2¹+0×2º，在运算中2的指数与位置数相对应，当左移五位后就变成了 1×2¹⁺⁵+0×2º⁺⁵= 1×2¹×2⁵+0×2º×2⁵ = (1×2¹+0×2º)×2⁵。所以由此可以看出当2左移五位就变成了 2×2⁵=64。<br>所以有一个数左移 n 为，即为这个数乘以2的n次方。<code>x<<n === x*2ⁿ</code>。<br>同样，左移运算也可以做向下取整运算，因为当 x 为整数时有 <code>(x<<0)===x</code>，所以当 x 为小数时有 <code>(x<<0)===Math.floor(x)</code>。</p><h4 id="gt-gt-有符号右移运算"><a href="#gt-gt-有符号右移运算" class="headerlink" title=">>有符号右移运算"></a>>>有符号右移运算</h4><p>它把 32 位数字中的所有数位整体右移，同时保留该数的符号（正号或负号）。有符号右移运算符恰好与左移运算相反。例如，把 64 右移 5 位，将变为 2。<br>因为有符号右移运算符与左移运算相反，所以有一个数左移 n 为，即为这个数除以2的n次方。<code>x<<n === x/2ⁿ</code>。<br>同样，有符号右移运算也可以做向下取整运算，因为当 x 为整数时有 <code>(x>>0)===x</code>，所以当 x 为小数时有 <code>(x>>0)===Math.floor(x)</code>。</p><h4 id="gt-gt-gt-无符号右移运算"><a href="#gt-gt-gt-无符号右移运算" class="headerlink" title=">>>无符号右移运算"></a>>>>无符号右移运算</h4><p>它将无符号 32 位数的所有数位整体右移。对于正数，无符号右移运算的结果与有符号右移运算一样，而负数则被作为正数来处理。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">-9</span> (base <span class="number">10</span>): <span class="number">11111111111111111111111111110111</span> (base <span class="number">2</span>)</span><br><span class="line"> --------------------------------</span><br><span class="line"><span class="number">-9</span> >>> <span class="number">2</span> (base <span class="number">10</span>): <span class="number">00111111111111111111111111111101</span> (base <span class="number">2</span>) = <span class="number">1073741821</span> (base <span class="number">10</span>)</span><br></pre></td></tr></table></figure></p><p>根据无符号右移的正数右移与有符号右移运算一样，而负数的无符号右移一定为非负的特征，可以用来判断数字的正负，如下：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">isPos</span>(<span class="params">n</span>) </span>{</span><br><span class="line"> <span class="keyword">return</span> (n === (n >>> <span class="number">0</span>)) ? <span class="literal">true</span> : <span class="literal">false</span>; </span><br><span class="line">}</span><br><span class="line"> </span><br><span class="line">isPos(<span class="number">-1</span>); <span class="comment">// false</span></span><br><span class="line">isPos(<span class="number">1</span>); <span class="comment">// true</span></span><br></pre></td></tr></table></figure></p><h3 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h3><p>根据 JS 的位运算，可以得出如下信息：<br>1、所有的位运算都可以对小数取底。<br>2、对于按位与<code>&</code>，可以用 <code>(n & 1) === 0 //true 为偶数，false 为奇数。</code>来判断奇偶。用<code>x&-1===Math.floor(x)</code>来向下取底。<br>3、对于按位或<code>|</code>，可以用<code>x|0===Math.floor(x)</code>来向下取底。<br>4、对于取反运算<code>~</code>，可以用<code>~~x===Math.floor(x)</code>来向下取底。<br>5、对于异或运算<code>^</code>，可以用<code>(x^0)===Math.floor(x)</code>来向下取底。<br>6、对于左移运算<code><<</code>，可以<code>x<<n === x*2ⁿ</code>来求2的n次方，用<code>x<<0===Math.floor(x)</code>来向下取底。<br>7、对于有符号右移运算<code>>></code>，可以<code>x<<n === x/2ⁿ</code>求一个数字的 N 等分，用<code>x>>0===Math.floor(x)</code>来向下取底。<br>8、对于无符号右移运算<code>>>></code>，可以<code>(n === (n >>> 0)) ? true : false;</code>来判断数字正负，用<code>x>>>0===Math.floor(x)</code>来向下取底。</p><p>用移位运算来替代普通算术能获得更高的效率。移位运算翻译成机器码的长度更短，执行更快，需要的硬件开销更小。</p>]]></content>
<summary type="html">
<p>这个话题的由来是2016年3月份的时候 <a href="https://www.npmjs.com/" target="_blank" rel="noopener">NPM</a> 社区发生了‘left-pad’事件，不久后社区就有人发布了用来补救的，也是现在大家能用到的 <a href="https://www.npmjs.com/package/left-pad" target="_blank" rel="noopener">left-pad</a> 库。</p>
</summary>
<category term="JavaScript" scheme="http://liu-xin.me/tags/JavaScript/"/>
</entry>
<entry>
<title>通过单元测试为API自动生成文档</title>
<link href="http://liu-xin.me/2017/01/03/%E9%80%9A%E8%BF%87%E5%8D%95%E5%85%83%E6%B5%8B%E8%AF%95%E4%B8%BAAPI%E8%87%AA%E5%8A%A8%E7%94%9F%E6%88%90%E6%96%87%E6%A1%A3/"/>
<id>http://liu-xin.me/2017/01/03/通过单元测试为API自动生成文档/</id>
<published>2017-01-03T15:10:26.000Z</published>
<updated>2021-01-08T15:02:37.868Z</updated>
<content type="html"><![CDATA[<p>在开发中，为项目生成文档是很常见的需求，很多第三方库（如<a href="https://www.npmjs.com/package/jsdoc" target="_blank" rel="noopener">jsdoc</a>、<a href="https://www.npmjs.com/package/swagger" target="_blank" rel="noopener">swagger</a>等）的做法是为需要生成文档的函数编写相应的符合规范的注释，然后运行相应的命令，生成一个静态网页形式的文档。<br><a id="more"></a></p><p>用注释生成文档的好处是可以为无论是普通函数还是 API，只要编写了相应的注释都能生成相应的文档，然而这种做法总觉得有点繁琐，尤其是只需要为 API 生成文档的时候，需要手动编写大量的输入和输出作为使用示例。而且我只想需要 markdown 形式的文档，丢在内部 Gitlab 的 wiki 上供前端人员査阅，然后可以根据 commit 的 history 查阅不同的版本。</p><p>不想手动为 API 文档编写大量的输入输出，那哪里会有输入输出呢，就很容易的想到了单元测试会产生输入和输出。好，那就用单元测试来为 API 生成文档。</p><p>我在单元测试中主要用的库有 <a href="https://www.npmjs.com/package/mocha" target="_blank" rel="noopener">mocha</a>、<a href="https://www.npmjs.com/package/supertest" target="_blank" rel="noopener">supertest</a>和<a href="https://www.npmjs.com/package/power-assert" target="_blank" rel="noopener">power-assert</a>，Web 框架为 <a href="https://www.npmjs.com/package/express" target="_blank" rel="noopener">express</a>。</p><p>完整代码示例：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// app.js</span></span><br><span class="line"><span class="keyword">const</span> express = <span class="built_in">require</span>(<span class="string">'express'</span>);</span><br><span class="line"><span class="keyword">const</span> bodyParser = <span class="built_in">require</span>(<span class="string">'body-parser'</span>);</span><br><span class="line"><span class="keyword">const</span> app = express();</span><br><span class="line"> </span><br><span class="line">app.use(bodyParser.json());</span><br><span class="line">app.use(bodyParser.urlencoded({ <span class="attr">extended</span>: <span class="literal">true</span> }));</span><br><span class="line"> </span><br><span class="line">app.post(<span class="string">'/user/create'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">req, res</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> name = req.body.name;</span><br><span class="line"> res.json({</span><br><span class="line"> status: <span class="number">200</span>,</span><br><span class="line"> error_code: <span class="number">0</span>,</span><br><span class="line"> message: <span class="string">'success'</span>,</span><br><span class="line"> data: {</span><br><span class="line"> id: <span class="string">'123abc'</span>,</span><br><span class="line"> name: <span class="string">'node'</span>,</span><br><span class="line"> gender: <span class="string">'male'</span>,</span><br><span class="line"> age: <span class="number">23</span>,</span><br><span class="line"> },</span><br><span class="line"> });</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line">app.get(<span class="string">'/user/search'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">req, res</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> name = req.query.name;</span><br><span class="line"> res.json({</span><br><span class="line"> status: <span class="number">200</span>,</span><br><span class="line"> error_code: <span class="number">0</span>,</span><br><span class="line"> message: <span class="string">'success'</span>,</span><br><span class="line"> data: {</span><br><span class="line"> id: <span class="string">'123abc'</span>,</span><br><span class="line"> name: <span class="string">'node'</span>,</span><br><span class="line"> gender: <span class="string">'male'</span>,</span><br><span class="line"> age: <span class="number">23</span>,</span><br><span class="line"> },</span><br><span class="line"> });</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line">app.get(<span class="string">'/user/:id'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">req, res</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> userId = req.params.id;</span><br><span class="line"> res.json({</span><br><span class="line"> status: <span class="number">200</span>,</span><br><span class="line"> error_code: <span class="number">0</span>,</span><br><span class="line"> message: <span class="string">'success'</span>,</span><br><span class="line"> data: {</span><br><span class="line"> id: <span class="string">'123abc'</span>,</span><br><span class="line"> name: <span class="string">'node'</span>,</span><br><span class="line"> gender: <span class="string">'male'</span>,</span><br><span class="line"> age: <span class="number">23</span>,</span><br><span class="line"> },</span><br><span class="line"> });</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line">app.listen(<span class="number">3000</span>, <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">`Server is listening on 3000`</span>);</span><br><span class="line">});</span><br><span class="line"><span class="built_in">module</span>.exports = app;</span><br></pre></td></tr></table></figure></p><p>下面是测试文件。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// test/uset.test.js</span></span><br><span class="line"><span class="keyword">const</span> assert = <span class="built_in">require</span>(<span class="string">'power-assert'</span>);</span><br><span class="line"><span class="keyword">const</span> supertest = <span class="built_in">require</span>(<span class="string">'supertest'</span>);</span><br><span class="line"><span class="keyword">const</span> U = <span class="built_in">require</span>(<span class="string">'../utils'</span>);</span><br><span class="line"><span class="keyword">const</span> app = <span class="built_in">require</span>(<span class="string">'../app'</span>);</span><br><span class="line"><span class="keyword">const</span> agent = supertest.agent(app);</span><br><span class="line"> </span><br><span class="line">describe(<span class="string">'Test'</span>, <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> it(<span class="string">'should create user success'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">done</span>) </span>{</span><br><span class="line"> U.test({</span><br><span class="line"> agent,</span><br><span class="line"> file: <span class="string">'user'</span>,</span><br><span class="line"> group: <span class="string">'用户相关API'</span>,</span><br><span class="line"> title: <span class="string">'创建用户'</span>,</span><br><span class="line"> method: <span class="string">'post'</span>,</span><br><span class="line"> url: <span class="string">'/user/create'</span></span><br><span class="line"> params: {</span><br><span class="line"> name: { <span class="attr">value</span>: <span class="string">'node'</span>, <span class="attr">type</span>: <span class="string">'String'</span>, <span class="attr">required</span>: <span class="literal">true</span>, <span class="attr">desc</span>: <span class="string">'名称'</span> },</span><br><span class="line"> gender: { <span class="attr">value</span>: <span class="string">'male'</span>, <span class="attr">type</span>: <span class="string">'String'</span>, <span class="attr">required</span>: <span class="literal">false</span>, <span class="attr">desc</span>: <span class="string">'性别'</span> },</span><br><span class="line"> age: { <span class="attr">value</span>: <span class="number">23</span>, <span class="attr">type</span>: <span class="string">'Int'</span>, <span class="attr">required</span>: <span class="literal">false</span>, <span class="attr">desc</span>: <span class="string">''</span> },</span><br><span class="line"> },</span><br><span class="line"> headers: { <span class="attr">entrance</span>: <span class="string">'client'</span> },</span><br><span class="line"> expect: <span class="number">200</span>,</span><br><span class="line"> callback (err, res) {</span><br><span class="line"> <span class="keyword">if</span> (err) <span class="keyword">return</span> done(err);</span><br><span class="line"></span><br><span class="line"> assert(res.body.data.name === <span class="string">'node'</span>);</span><br><span class="line"> assert(res.body.data.age === <span class="number">23</span>);</span><br><span class="line"> done();</span><br><span class="line"> },</span><br><span class="line"> });</span><br><span class="line"> });</span><br><span class="line"> </span><br><span class="line"> it(<span class="string">'should search user success'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">done</span>) </span>{</span><br><span class="line"> U.test({</span><br><span class="line"> agent,</span><br><span class="line"> file: <span class="string">'user'</span>,</span><br><span class="line"> group: <span class="string">'用户相关API'</span>,</span><br><span class="line"> title: <span class="string">'搜索用户'</span>,</span><br><span class="line"> method: <span class="string">'get'</span>,</span><br><span class="line"> url: <span class="string">'/user/search'</span></span><br><span class="line"> params: {</span><br><span class="line"> name: { <span class="attr">value</span>: <span class="string">'node'</span>, <span class="attr">type</span>: <span class="string">'String'</span>, <span class="attr">required</span>: <span class="literal">true</span>, <span class="attr">desc</span>: <span class="string">'名称'</span> },</span><br><span class="line"> },</span><br><span class="line"> expect: <span class="number">200</span>,</span><br><span class="line"> callback (err, res) {</span><br><span class="line"> <span class="keyword">if</span> (err) <span class="keyword">return</span> done(err);</span><br><span class="line"> </span><br><span class="line"> assert(res.body.data.name === <span class="string">'node'</span>);</span><br><span class="line"> assert(res.body.data.age === <span class="number">23</span>);</span><br><span class="line"> done();</span><br><span class="line"> },</span><br><span class="line"> });</span><br><span class="line"> });</span><br><span class="line"> </span><br><span class="line"> it(<span class="string">'should search user success'</span>, <span class="function"><span class="keyword">function</span> (<span class="params">done</span>) </span>{</span><br><span class="line"> U.test({</span><br><span class="line"> agent,</span><br><span class="line"> file: <span class="string">'user'</span>,</span><br><span class="line"> group: <span class="string">'用户相关API'</span>,</span><br><span class="line"> title: <span class="string">'获取用户信息'</span>,</span><br><span class="line"> method: <span class="string">'get'</span>,</span><br><span class="line"> url: <span class="string">'/user/:id'</span></span><br><span class="line"> params: {</span><br><span class="line"> id: { <span class="attr">value</span>: <span class="string">'123abc'</span>, <span class="attr">type</span>: <span class="string">'String'</span>, <span class="attr">required</span>: <span class="literal">true</span>, <span class="attr">desc</span>: <span class="string">''</span> },</span><br><span class="line"> },</span><br><span class="line"> expect: <span class="number">200</span>,</span><br><span class="line"> callback (err, res) {</span><br><span class="line"> <span class="keyword">if</span> (err) <span class="keyword">return</span> done(err);</span><br><span class="line"> </span><br><span class="line"> assert(res.body.data.name === <span class="string">'node'</span>);</span><br><span class="line"> assert(res.body.data.age === <span class="number">23</span>);</span><br><span class="line"> done();</span><br><span class="line"> },</span><br><span class="line"> });</span><br><span class="line"> });</span><br><span class="line">});</span><br></pre></td></tr></table></figure></p><p>在测试文件中，测试用例的代码调用到了 utils.js 中的 test 方法，该方法的主要作用是接收单元测试的输入和输出并生成相应的文档，其中需要向 test 方法传入一个对象作为参数，对象中的字段解读如下：</p><blockquote><p>agent：调用 API 的代理。<br>file：生成的文档的文件名称。<br>group：某一组文档的名称。<br>title：接口的名称。<br>method：接口的方法。<br>params：接口的参数，即输入。<br>headers: 添加到请求头中的信息。<br>expect：supertest 的expect。<br>callback：supertest 方法的 end 的回调函数。</p></blockquote><p>utils.js代码如下：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// utils.js</span></span><br><span class="line"><span class="keyword">const</span> path = <span class="built_in">require</span>(<span class="string">'path'</span>);</span><br><span class="line"><span class="keyword">const</span> fs = <span class="built_in">require</span>(<span class="string">'fs'</span>);</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> mdStr = {};</span><br><span class="line">exports.test = <span class="function"><span class="keyword">function</span> (<span class="params">obj</span>) </span>{</span><br><span class="line"> <span class="keyword">if</span> (!mdStr[obj.group]) {</span><br><span class="line"> mdStr[obj.group] = <span class="string">''</span>;</span><br><span class="line"> mdStr[obj.group] += <span class="string">'## '</span> + obj.group + <span class="string">'\n\n'</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">const</span> fields = {};</span><br><span class="line"> </span><br><span class="line"> mdStr[obj.group] += <span class="string">`### <span class="subst">${ obj.title }</span> \`<span class="subst">${ obj.method }</span>\` <span class="subst">${ obj.url }</span> \n\n#### 参数\n`</span>;</span><br><span class="line"> mdStr[obj.group] += <span class="string">'\n参数名 | 类型 | 是否必填 | 说明\n-----|-----|-----|-----\n'</span>;</span><br><span class="line"> <span class="built_in">Object</span>.keys(obj.params).forEach(<span class="function"><span class="keyword">function</span> (<span class="params">param</span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> paramVal = obj.params[param];</span><br><span class="line"> fields[param] = paramVal[<span class="string">'value'</span>];</span><br><span class="line"> mdStr[obj.group] += <span class="string">`<span class="subst">${ param }</span> | <span class="subst">${ paramVal[<span class="string">'type'</span>] }</span> | <span class="subst">${ paramVal[<span class="string">'required'</span>] ? <span class="string">'是'</span> : <span class="string">'否'</span> }</span> | <span class="subst">${ paramVal[<span class="string">'desc'</span>] }</span> \n`</span>;</span><br><span class="line"> });</span><br><span class="line"> mdStr[obj.group] += <span class="string">'\n#### 使用示例\n\n请求参数: \n\n'</span>;</span><br><span class="line"> </span><br><span class="line"> mdStr[obj.group] += <span class="string">'```json\n'</span> + <span class="built_in">JSON</span>.stringify(fields, <span class="literal">null</span>, <span class="number">2</span>) + <span class="string">'\n```\n'</span>;</span><br><span class="line"> mdStr[obj.group] += <span class="string">'\n返回结果:\n\n'</span>;</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">if</span> (obj.url.indexOf(<span class="string">':'</span>) > <span class="number">-1</span>) {</span><br><span class="line"> obj.url = obj.url.replace(<span class="regexp">/:\w*/g</span>, <span class="function"><span class="keyword">function</span> (<span class="params">word</span>) </span>{</span><br><span class="line"> <span class="keyword">return</span> fields[word.substr(<span class="number">1</span>)];</span><br><span class="line"> });</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> obj.agent[obj.method](obj.url)</span><br><span class="line"> .set(obj.header || {})</span><br><span class="line"> .query(fields)</span><br><span class="line"> .send(fields)</span><br><span class="line"> .expect(obj.expect)</span><br><span class="line"> .end(<span class="function"><span class="keyword">function</span> (<span class="params">err, res</span>) </span>{</span><br><span class="line"> mdStr[obj.group] += <span class="string">'```json\n'</span> + <span class="built_in">JSON</span>.stringify(res.body, <span class="literal">null</span>, <span class="number">2</span>) + <span class="string">'\n```\n'</span>;</span><br><span class="line"> mdStr[obj.group] += <span class="string">'\n'</span>;</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">if</span> (process.env[<span class="string">'GEN_DOC'</span>] > <span class="number">0</span>) {</span><br><span class="line"> fs.writeFileSync(path.resolve(__dirname, <span class="string">'./docs/'</span>, obj.file + <span class="string">'.md'</span>), mdStr[obj.group]);</span><br><span class="line"> }</span><br><span class="line"> obj.callback(err, res);</span><br><span class="line"> });</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>这样，在根目录创建一个 docs 目录，运行 <code>npm run test:doc</code> 命令，就会在 docs 目录下生成文档。如果运行单元测试不想生成文档，直接用<code>npm test</code>就可以了，相应的package.json配置如下：<br><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">"scripts": {</span><br><span class="line"> "test": "export NODE_ENV='test' && mocha",</span><br><span class="line"> "test:doc": "export NODE_ENV='test' && export GEN_DOC=1 && mocha"</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>如果不想为某个 API 生成文档，就不要调用 utils 的 test，直接按原生的写法就可以了。</p><p>若需要对参数进行签名，可在调用 test 方法时，增加形如<code>sign: true</code>的配置，然后在 test 方法中做相应的判断和实现相应的签名。<br>生成的文档内容形式如下：<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/test_doc.jpg" alt="test_doc"></p>]]></content>
<summary type="html">
<p>在开发中，为项目生成文档是很常见的需求，很多第三方库（如<a href="https://www.npmjs.com/package/jsdoc" target="_blank" rel="noopener">jsdoc</a>、<a href="https://www.npmjs.com/package/swagger" target="_blank" rel="noopener">swagger</a>等）的做法是为需要生成文档的函数编写相应的符合规范的注释，然后运行相应的命令，生成一个静态网页形式的文档。<br>
</summary>
<category term="doc" scheme="http://liu-xin.me/tags/doc/"/>
</entry>
<entry>
<title>利用redis将log4js产生的日志送到logstash</title>
<link href="http://liu-xin.me/2016/08/23/%E5%88%A9%E7%94%A8redis%E5%B0%86log4js%E4%BA%A7%E7%94%9F%E7%9A%84%E6%97%A5%E5%BF%97%E9%80%81%E5%88%B0logstash/"/>
<id>http://liu-xin.me/2016/08/23/利用redis将log4js产生的日志送到logstash/</id>
<published>2016-08-23T08:43:51.000Z</published>
<updated>2021-01-08T15:05:57.986Z</updated>
<content type="html"><![CDATA[<p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/log4js-logstash-redis.png" alt="log4js-logstash-redis"></p><p>在用Node.js开发项目的时候，我们常用 <a href="https://www.npmjs.com/package/log4js" target="_blank" rel="noopener">log4js</a> 模块来进行日志的记录，可以通过配置 log4js 的 <a href="https://github.com/nomiddlename/log4js-node/wiki/Appenders" target="_blank" rel="noopener">Appenders</a> 将日志输出到Console、File和GELF等不同的地方。<br><a id="more"></a></p><h2 id="logstash"><a href="#logstash" class="headerlink" title="logstash"></a>logstash</h2><p>logstash 是 <a href="https://www.elastic.co/" target="_blank" rel="noopener">elastic</a> 技术栈的其中一员，常被用来收集和解析日志。一种比较常见的做法就是在项目工程中把日志写入到日志文件中，然后用 logstash 读取和解析日志文件。比如在 Node.js 项目中，若要将日志记录到文件中，只需对 log4js 做如下配置即可：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> log4js = <span class="built_in">require</span>(<span class="string">'log4js'</span>);</span><br><span class="line">log4js.configure({</span><br><span class="line"> appenders: [{</span><br><span class="line"> type: <span class="string">'file'</span>,</span><br><span class="line"> filename: <span class="string">'./example.log'</span></span><br><span class="line"> }]</span><br><span class="line">});</span><br><span class="line"> </span><br><span class="line"><span class="keyword">const</span> logger = log4js.getLogger();</span><br><span class="line">logger.info(<span class="string">'hello'</span>);</span><br></pre></td></tr></table></figure></p><p>这样日志就会被记录到 <code>example.log</code> 文件中，然后再对 logstash 的 input 做如下配置：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">file {</span><br><span class="line"> path => "YourLogPath/example.log"</span><br><span class="line"> start_position => "beginning"</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>这样 logstash 就可以读取这个日志文件。可是这样总感觉用一个文件作为中转略显麻烦，如果能将 log4js 产生的日志直接送到 logstash 就更好了。</p><h2 id="logstashUDP"><a href="#logstashUDP" class="headerlink" title="logstashUDP"></a>logstashUDP</h2><p>log4js 内置了 logstashUDP 来直接将日志输出到 logstash。配置如下：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">log4js.configure({</span><br><span class="line"> appenders: [{</span><br><span class="line"> type: <span class="string">"logstashUDP"</span>,</span><br><span class="line"> host: <span class="string">"localhost"</span>,</span><br><span class="line"> port: <span class="number">12345</span></span><br><span class="line">}]</span><br><span class="line">});</span><br></pre></td></tr></table></figure></p><p>然后将 logstash 的配置改成下面这样：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">input {</span><br><span class="line">udp {</span><br><span class="line"> host => "127.0.0.1"</span><br><span class="line">port => 12345</span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>嗯，很简单嘛！现在 log4js 产生的日志就能直接送到 logstash 了，而不再需要用一个文件作为中转。但是，当我在使用的时候发现一个问题，就是如果 logstash 服务挂了，这时候 log4js 仍然在不断的产生日志数据，这时候首先想到的就是把 logstash 重新启动起来，但重启后却发现 logstash 没能获取到在 logstash 挂掉的时候 log4js 产生的数据，也就是说如果 logstash 挂掉了的话，那么 log4js 产生的数据就会丢失，不会被处理。<br><br>这时候就想着用一个代理来暂存 log4js 产生的数据，log4js 将数据输出到代理，logstash 从代理那里读取数据，logstash 读取一条数据，代理就丢弃掉那条数据。对，也就是队列。这样就不会有数据丢失的问题了。</p><h2 id="log4js-logstash-redis"><a href="#log4js-logstash-redis" class="headerlink" title="log4js-logstash-redis"></a>log4js-logstash-redis</h2><p><a href="https://www.npmjs.com/package/log4js-logstash-redis" target="_blank" rel="noopener">log4js-logstash-redis</a> 就是为了解决上述问题而开发的一个 log4js 的 Appender。<br><br>他的原理是：log4js 将产生的日志输出到 redis，然后让 logstash 从 redis 读取数据。但是直接叫 log4js-redis 就好了嘛，为什么叫 log4js-logstash-redis 呢？这是因为将日志格式针对 logstash 做过一些更友好的定制。😊 <br></p><h3 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">npm install log4js-logstash-redis --save</span><br></pre></td></tr></table></figure><h3 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h3><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">log4js.configure({</span><br><span class="line"> appenders: [{</span><br><span class="line"> type: <span class="string">"log4js-logstash-redis"</span>,</span><br><span class="line"> key: <span class="string">"test"</span>,</span><br><span class="line"> redis: {</span><br><span class="line"> db: <span class="number">1</span></span><br><span class="line"> }</span><br><span class="line">}]</span><br><span class="line">});</span><br></pre></td></tr></table></figure><p>redis 配置可选，没有的话 redis 连接就采用默认值。logstash 的配置如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">redis {</span><br><span class="line">data_type => "list"</span><br><span class="line">key => "test"</span><br><span class="line">codec => json</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>其中 data_type 的值一定要是 list。</p>]]></content>
<summary type="html">
<p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/log4js-logstash-redis.png" alt="log4js-logstash-redis"></p>
<p>在用Node.js开发项目的时候，我们常用 <a href="https://www.npmjs.com/package/log4js" target="_blank" rel="noopener">log4js</a> 模块来进行日志的记录，可以通过配置 log4js 的 <a href="https://github.com/nomiddlename/log4js-node/wiki/Appenders" target="_blank" rel="noopener">Appenders</a> 将日志输出到Console、File和GELF等不同的地方。<br>
</summary>
<category term="redis" scheme="http://liu-xin.me/tags/redis/"/>
<category term="logstash" scheme="http://liu-xin.me/tags/logstash/"/>
</entry>
<entry>
<title>用ELK处理日志时碰到的几个问题</title>
<link href="http://liu-xin.me/2016/08/16/%E7%94%A8ELK%E5%A4%84%E7%90%86%E6%97%A5%E5%BF%97%E6%97%B6%E7%A2%B0%E5%88%B0%E7%9A%84%E5%87%A0%E4%B8%AA%E9%97%AE%E9%A2%98/"/>
<id>http://liu-xin.me/2016/08/16/用ELK处理日志时碰到的几个问题/</id>
<published>2016-08-16T03:50:55.000Z</published>
<updated>2021-01-08T15:07:19.644Z</updated>
<content type="html"><![CDATA[<p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/elk.png" alt="elk"><br>常说的ELK是指包括 elasticsearch、logstash 和 kibana 的一套技术栈，常用来处理日志等数据。最近在用这一套 stack 处理 nginx 的日志，而且还用到了 elastic 的另一个产品 beats，来作为客户端 shipper，形成了ELKB Stack。<br><a id="more"></a><br><br><br>在使用这一套技术栈的时候碰到了一些其实并不高深的小问题，在这里写下来作为一个记录。</p><h2 id="日志格式"><a href="#日志格式" class="headerlink" title="日志格式"></a>日志格式</h2><p>常见的 nginx 的 access_log 日志都是用竖线<code>|</code>作为分隔，但我们的是用逗号<code>,</code>作为分隔，就像下面这样：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">04/Aug/2016:15:37:09 +0800,112.124.127.44,"-","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36",GET,/,0.003,0.003,146,199,23025,1,200,200,14,"-",TLSv1,AES256-SHA,127.0.0.1:8384</span><br></pre></td></tr></table></figure></p><p>一开始我并没意识到这会存在什么问题，就在 logstash 的配置文件中用 filter 里的 ruby 来解析，如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">ruby {</span><br><span class="line"> init => "@kname = ['time_local', 'remote_addr', 'remote_user', 'http_user_agent', 'request_method', 'uri', 'request_time', 'upstream_response_time', 'request_length', 'bytes_sent', 'connection', 'connection_request', 'status', 'upstream_status', 'body_bytes_sent', 'http_referer', 'ssl_protocol', 'ssl_cipher', 'upstream_addr']"</span><br><span class="line"> code => "</span><br><span class="line"> new_event = LogStash::Event.new(Hash[@kname.zip(event['message'].split(','))])</span><br><span class="line"> new_event.remove('@timestamp')</span><br><span class="line"> event.append(new_event)</span><br><span class="line"> "</span><br><span class="line"> remove_field => ["message"]</span><br><span class="line"> }</span><br></pre></td></tr></table></figure></p><p>无非就是把<code>split('|')</code>换成<code>split(',')</code>嘛，很简单！但是当我在终端观察输出的时候，发现却是这样的。<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/elk-1.png" alt="elk-1"><br>User-Agent 被分割成了两个字段的内容，导致从<code>request_method</code>字段开始，字段名与其对应的值都错位了，原来在日志内容中的浏览器的User-Agent <code>"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"</code>也包含逗号<code>,</code>导致它也被分割了，以致后面的内容都错位了，很明显这种情况不应该被分割。<br><br>由于对 ruby 不是很熟悉，就没有继续在ruby filter中想办法来解决这个问题，我就继续查阅了 Logstash 其他 filter 的用法，发现用 grok filter 似乎可以解决这个问题，于是就用 grok 替换了 ruby，配置如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">grok {</span><br><span class="line"> match=>{"message" => "%{GREEDYDATA:time_local},%{IP:remote_addr},\"%{GREEDYDATA:remote_user}\",\"%{GREEDYDATA:http_user_agent}\",</span><br><span class="line"> %{WORD:request_method},%{URIPATHPARAM:uri},%{GREEDYDATA:request_time},%{GREEDYDATA:upstream_response_time},%{NUMBER:request_length},%{NUMBER:bytes_sent},</span><br><span class="line"> %{NUMBER:connection},%{NUMBER:connection_request},%{NUMBER:status},%{NUMBER:upstream_status},%{NUMBER:body_bytes_sent},</span><br><span class="line"> \"%{GREEDYDATA:http_referer}\",%{GREEDYDATA:ssl_protocol},%{GREEDYDATA:ssl_cipher},%{GREEDYDATA:upstream_addr}"}</span><br><span class="line"> }</span><br></pre></td></tr></table></figure></p><p>现在再重新解析观察一节结果。<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/elk-2.png" alt="elk-2"><br>嗯，问题解决，得到了正确的解析结果。但是，总觉得这样太不优雅了，看着无比的臃肿，而且这种正则匹配效率也不是很好。后来经同事提醒，csv filter 可以很好的解决这个问题，在查阅了csv filter的用法后，将 grok 替换成如下配置：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">csv {</span><br><span class="line">source => "message"</span><br><span class="line">columns => ["time_local", "remote_addr", "remote_user", "http_user_agent", "request_method", "uri", "request_time", "upstream_response_time", "request_length", "bytes_sent", "connection", "connection_request", "status", "upstream_status", "body_bytes_sent", "http_referer", "ssl_protocol", "ssl_cipher", "upstream_addr"]</span><br><span class="line">convert => {</span><br><span class="line">"request_time" => "float"</span><br><span class="line">"upstream_response_time" => "float"</span><br><span class="line">"connection" => "integer"</span><br><span class="line">"connection_request" => "integer"</span><br><span class="line">"bytes_sent" => "integer"</span><br><span class="line">"body_bytes_sent" => "integer"</span><br><span class="line">"request_length" => "integer"</span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>OK，试了一下，完美解决，既优雅又正确。</p><h2 id="geo-point"><a href="#geo-point" class="headerlink" title="geo_point"></a>geo_point</h2><p>geo_point 是 elasticsearch 中的一种数据类型，Kibana 根据类型为 geo_point 的字段生成 Geo Coordinates。但是当我点Geo Coordinates得到的确实如下图的提示信息：<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/elk-3.png" alt="elk-3"><br>在我的nginx-test1_log（nginx-test1_log为我用logstash将ngix日志导入到 elasticsearch 中取的索引名。）索引中不能找到类型geo_point类型的字段。<br><br>Logstash 中有一个geoip filter 是专门来处理将 ip 解析成地理坐标的，解析完后会得到location字段，我的配置如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">geoip {</span><br><span class="line">source => remote_addr</span><br><span class="line">fields => ["city_name", "country_name", "latitude", "longitude", "real_region_name", "location"]</span><br><span class="line">remove_field => ["[geoip][latitude]", "[geoip][longitude]"]</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>Kibana 通常用那个location字段来生成 Geo Coordinates，于是我就去查看 nginx-test1_log 的mapping（通过 <a href="http://localhost:9200/nginx-test1_log/_mapping" target="_blank" rel="noopener">http://localhost:9200/nginx-test1_log/_mapping</a> 查看。）发现location的类型是double，并不是geo_point，于是就想着把location字段的类型改为geo_point，然而发现并不能或者特别麻烦，就放弃了这个想法，Google 了一圈后，发现是 elasticsearch （以下简称『es』） 的 template 造成的（可通过 <a href="http://localhost:9200/_template" target="_blank" rel="noopener">http://localhost:9200/_template</a> 查看），es 会对 logstash 导入的数据采用默认的 logstash template，这个模板就通过 mapping 指定了location的类型为geo_point，但是如果在 Logstash 的 output 中为 es 指定的index的名字不是以logstash开头话，就不会采用这个模板，相应的location的类型就不会是geo_point了，由于我这里就是以nginx开头，所以就出现了这个问题。解决办法就是自己创建一个 template，然后在输出时指定，我直接把 logstash template 复制过来然后调整了下名字，调整后的 logstash 输出配置如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">output {</span><br><span class="line"> elasticsearch {</span><br><span class="line"> hosts => ["0.0.0.0:9200"]</span><br><span class="line"> index => "nginx-%{source_tag}"</span><br><span class="line"> template => "/home/liuxin/logstash/templates/nginx-template.json"</span><br><span class="line"> template_name => "nginx-*"</span><br><span class="line"> template_overwrite => true</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>至此，问题就解决了。<br><br>入门ELK不久，感觉ELK高效、简单、易用和生态丰富，以后肯定还会碰到很多问题，再陆陆续续补充。</p>]]></content>
<summary type="html">
<p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/elk.png" alt="elk"><br>常说的ELK是指包括 elasticsearch、logstash 和 kibana 的一套技术栈，常用来处理日志等数据。最近在用这一套 stack 处理 nginx 的日志，而且还用到了 elastic 的另一个产品 beats，来作为客户端 shipper，形成了ELKB Stack。<br>
</summary>
<category term="logstash" scheme="http://liu-xin.me/tags/logstash/"/>
<category term="日志" scheme="http://liu-xin.me/tags/%E6%97%A5%E5%BF%97/"/>
<category term="ELK" scheme="http://liu-xin.me/tags/ELK/"/>
<category term="elasticsearch" scheme="http://liu-xin.me/tags/elasticsearch/"/>
<category term="kibana" scheme="http://liu-xin.me/tags/kibana/"/>
<category term="大数据" scheme="http://liu-xin.me/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
</entry>
<entry>
<title>利用Jenkins和Docker做持续集成</title>
<link href="http://liu-xin.me/2016/07/13/%E5%88%A9%E7%94%A8Jenkins%E5%92%8CDocker%E5%81%9A%E6%8C%81%E7%BB%AD%E9%9B%86%E6%88%90/"/>
<id>http://liu-xin.me/2016/07/13/利用Jenkins和Docker做持续集成/</id>
<published>2016-07-13T07:12:16.000Z</published>
<updated>2021-01-08T15:09:04.298Z</updated>
<content type="html"><![CDATA[<p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/docker_jenkins.jpg" alt="docker_jenkins"><br>持续集成（Continuous integration，简称CI）是一套标准的互联网软件开发和发布流程，主要指频繁的将代码集成到主干，让产品可以快速迭代，同时能保证高质量。<br><a id="more"></a></p><h2 id="什么是Jenkins"><a href="#什么是Jenkins" class="headerlink" title="什么是Jenkins"></a>什么是Jenkins</h2><p>Jenkins是一款由Java开发的开源软件项目，旨在提供一个开放易用的软件平台，使持续集成变成可能。<br><br>利用Jenkins持续集成Node.js项目之后，就不用每次都登录到服务器，执行pm2 restart xxx或者更原始一点的kill xx，然后node xxx。通过Jenkins，只需单击『立即构建』按钮，就可以自动从Git仓库拉取代码，然后部署到远程服务器，执行一些安装依赖包和测试的命令，最后启动应用。如果需要部署到多台服务器上，只需在Jenkins上多配置相应的服务器数量，就可以通过Jenkins部署到多台服务器上。</p><h2 id="通过Docker安装和启动Jenkins"><a href="#通过Docker安装和启动Jenkins" class="headerlink" title="通过Docker安装和启动Jenkins"></a>通过Docker安装和启动Jenkins</h2><p>Jenkins需要Java环境，有了Docker这个利器，我们就省去了安装Java环境的麻烦，只需执行如下命令即可。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker pull jenkins:latest</span><br></pre></td></tr></table></figure></p><p>有一个通常的做法是要将Jenkins文件存储地址挂载到主机上，万一Jenkins的服务器重装或迁移，可以很方便的把之前的项目配置保留。所用可通过如下命令来启动Jenkins容器。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker run -d --name myjenkins -p 49001:8080 -v ${pwd}/data:/var/jenkins_home jenkins</span><br></pre></td></tr></table></figure></p><p>上面安装和启动Jenkins容器的做法，常常会出现错误，错误日志如下。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">touch: cannot touch ‘/var/jenkins_home/copy_reference_file.log’: Permission denied</span><br><span class="line">Can not write to /var/jenkins_home/copy_reference_file.log. Wrong volume permissions?</span><br></pre></td></tr></table></figure></p><p>这里推荐直接从 <a href="https://github.com/denverdino/docker-jenkins" target="_blank" rel="noopener">https://github.com/denverdino/docker-jenkins</a> 获得相关代码，并构建自己的Jenkins镜像。执行命令如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">git clone https://github.com/AliyunContainerService/docker-jenkins</span><br><span class="line">cd docker-jenkins/jenkins</span><br><span class="line">docker build -t myjenkins .</span><br></pre></td></tr></table></figure></p><p>然后基于新镜像启动Jenkins容器。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">docker rm -f jenkins</span><br><span class="line">docker run -d -p 8080:8080 -p 50000:50000 -v $(pwd)/data:/var/jenkins_home --name jenkins myjenkins</span><br></pre></td></tr></table></figure></p><p>用<code>docker-machine ip</code>获取到docker的IP后，用浏览器访问这个IP的8080端口，就会有如下图所示的Jenkins的主界面。<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/jenkins_homepage.jpg" alt="docker_homepage"></p><h2 id="通过Jenkins自动化部署Node-js项目"><a href="#通过Jenkins自动化部署Node-js项目" class="headerlink" title="通过Jenkins自动化部署Node.js项目"></a>通过Jenkins自动化部署Node.js项目</h2><p>要实现自动化部署，需要在Jenkins中安装Git Plugin和Publish Over SSH这连个插件。<br><br>插件安装完成后重启Jenkins，进入系统管理→系统设置来对插件进行简单的配置，增加远程的服务器配置。如下图所示，填入我们待部署的生产服务器的IP地址、SSH端口及用户名、密码等信息。如果远程服务是通过key来登录的，那么还需要把key的存放路径写上。<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/jenkins_remotehost.jpg" alt="docker_jenkins_ssh"><br>回到Jenkins主页，单击左上角的『新建』按钮就可以开启一个新项目，给项目起名node_test，选择创建一个自由风格的软件项目，单击「OK」按钮，就进入了此项目的创建页面。<br><br>在配置页，我们找到『源码管理』选项，配置好Github上的源码地址。<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/jenkins_git.jpg" alt="docker_github_code"><br>然后单击「Add」按钮，配置Github账号，Jenkins就是通过这个账号来拉取源代码的。<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/jenkins_git_account.png" alt="docker_github_account"><br>在『构建』一栏，单击下拉菜单，选择Execute shell，Jenkins从Github获取代码代码是自动执行的，这一步主要是将最新的代码打包，如下图。<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/execShell_1.png" alt="execShell"><br>将代码打包后需要把代码发送的远程的生产服务器上，这时需要选择「Send files…」选项，需要填入的指令如下图。<br><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/sendFile_1.png" alt="sendFile"><br>在Source files一栏中将刚刚打包的文件名node_test.tar.gz填入<br><br>在Remote directory一栏中，填写发送代码包的远程保存地址，我们在这里写入var/，在远程主机上代码包的路径就是/var/node_test.tar.gz<br><br>Exec Command一栏中的命令的主要作用是：</p><ol><li>删除之前的容器。</li><li>删除原来项目的代码。</li><li>解压新代码。</li><li>在容器中安装新代码的依赖包，然后将真个应用启动起来。</li><li>删除发送过来的代码包。<br><br></li></ol><p>至此项目配置完成，保存之后回到首页，单击左侧的『立即构建』按钮，如果小球变为蓝色的话就是构建成功，我们的项目也就部署成功了。</p><p><em>参考：<a href="https://yq.aliyun.com/articles/53990" target="_blank" rel="noopener">https://yq.aliyun.com/articles/53990</a></em>、<em>《Node.js实战（第二季）》</em></p>]]></content>
<summary type="html">
<p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/docker_jenkins.jpg" alt="docker_jenkins"><br>持续集成（Continuous integration，简称CI）是一套标准的互联网软件开发和发布流程，主要指频繁的将代码集成到主干，让产品可以快速迭代，同时能保证高质量。<br>
</summary>
<category term="Docker" scheme="http://liu-xin.me/tags/Docker/"/>
<category term="Jenkins" scheme="http://liu-xin.me/tags/Jenkins/"/>
</entry>
<entry>
<title>初探Docker</title>
<link href="http://liu-xin.me/2016/07/07/%E5%88%9D%E6%8E%A2Docker/"/>
<id>http://liu-xin.me/2016/07/07/初探Docker/</id>
<published>2016-07-07T08:57:28.000Z</published>
<updated>2021-01-08T15:02:01.958Z</updated>
<content type="html"><![CDATA[<p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/docker.jpg" alt="docker"><br>Docker让开发人员易于快速构建可随时运行的容器化应用程序，它大大简化了管理和部署应用程序的任务。下面主要介绍一些基本指令和用法，由于主要是在Mac上使用，所以有些东西在Mac上才会涉及到。<br><a id="more"></a></p><h2 id="安装（只适用于macOS）"><a href="#安装（只适用于macOS）" class="headerlink" title="安装（只适用于macOS）"></a>安装（只适用于macOS）</h2><ol><li>下载官网的DockerToolbox.pkg安装包安装，这里推荐在阿里云上的镜像<a href="http://mirrors.aliyun.com/docker-toolbox/mac/?spm=0.0.0.0.8EK1Sh" target="_blank" rel="noopener">http://mirrors.aliyun.com/docker-toolbox/mac/?spm=0.0.0.0.8EK1Sh</a> 。</li><li>使用brew安装<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">brew cask update</span><br><span class="line">brew cask install docker</span><br></pre></td></tr></table></figure></li></ol><h2 id="快速开始"><a href="#快速开始" class="headerlink" title="快速开始"></a>快速开始</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">docker-machine create --engine-registry-mirror=https://f9fd872q.mirror.aliyuncs.com -d virtualbox default</span><br><span class="line"># 创建一台安装有Docker环境的Linux虚拟机，指定机器名称为default，同时配置Docker加速器地址.</span><br></pre></td></tr></table></figure><p>记得一定要用加速器，这很重要！<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">docker-machine env default</span><br><span class="line">eval "$(docker-machine env default)"</span><br></pre></td></tr></table></figure></p><p>通过运行docker info指令，若能看到Containers、Running和Images等相关的信息，这就表明Docker环境搭建好了。</p><h2 id="一些指令"><a href="#一些指令" class="headerlink" title="一些指令"></a>一些指令</h2><p>docker ps <code>列出所有运行中的容器</code> <br><br>docker ps -a <code>列出所有容器</code> <br><br>docker images <code>列出镜像</code> <br><br>docker rmi [Image] <code>删除一个镜像</code> <br><br>docker rm [Container] <code>删除一个容器</code> <br><br>docker stop [Container] <code>停止一个正在运行的容器</code></p><p><strong>docker run [OPTIONS] IMAGE[:TAG] [COMMAND] [ARG…] 依附一个镜像运行一个容器</strong></p><p>OPTIONS：</p><ul><li>-d：分离模式，在后台运行容器，并且打印出容器ID</li><li>-i：保持输入流开放即使没有附加输入流</li><li>-t：分配一个伪造的终端输入</li><li>-p：匹配镜像内的网络端口号</li><li>-e：设置环境变量</li><li>-link：连接到另一个容器</li><li>-name：分配容器的名称，如果没有指定就会随机生成一个</li></ul><p>Examples：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">docker run -it ubuntu 启动一个依赖Ubuntu镜像并带有终端输入的容器</span><br><span class="line"></span><br><span class="line">docker run --name db -d -e MYSQL_ROOT_PASSWORD=123 -p 3306:3306 mysql:latest 启动一个依赖mysql镜像的容器，-name db将容器命名为db；-d在后台运行；</span><br><span class="line">-e MYSQL_ROOT_PASSWORD=123设置环境变量，参数告诉docker所提供的环境变量，这之后跟着的变量正是MySQL镜像检查且用来设置的默认root用户的密码；</span><br><span class="line">-p 3306:3306告诉引擎用户想要将容器内的3306端口映射到外部的3306端口上。</span><br><span class="line"></span><br><span class="line">docker exec -it db 告诉docker用户想要在名为db的容器里执行一个命令。</span><br><span class="line"></span><br><span class="line">docker run -it -p 8123:8123 --link db:db -e DATABASE_HOST=db node4 使用link参数告诉docker我们想要连接到另外的容器上，link db:db连接</span><br><span class="line">到名为db的容器上，并且用db指代该容器。</span><br></pre></td></tr></table></figure></p><h2 id="在工程中使用Docker"><a href="#在工程中使用Docker" class="headerlink" title="在工程中使用Docker"></a>在工程中使用Docker</h2><p>在项目工程中常常在根目录一个Dockerfile文件，用来构建项目的运行环境。下面是一个Node.js工程中Docerfile文件的示例。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">FROM node:4 #指定基础镜像</span><br><span class="line"></span><br><span class="line">ADD . /app #将当前目录下的所有东西拷贝到名为app/的容器目录里</span><br><span class="line"></span><br><span class="line">RUN cd /app;</span><br><span class="line">npm install --production 在镜像里运行命令</span><br><span class="line"></span><br><span class="line">EXPOSE 3000 #将服务的端口暴露出来</span><br><span class="line"></span><br><span class="line">CMD ["node"， "/app/index.js"] #运行服务</span><br></pre></td></tr></table></figure></p><p>有了这个Dockerfile文件后，只需在项目根目录执行docker build -t node4 .就可以在docker引擎中创建所需要的容器，并将代码放到容器中，-t node4表示使用标签node4标记该镜像，之后就可以使用这个标签来代指该镜像，.最后那个点表示在当前目录查找Dockerfile文件。<br><br>通过执行docker run -it -p 8123:8123 node4就可以将该容器运行起来了。</p><h2 id="Docker-Compose"><a href="#Docker-Compose" class="headerlink" title="Docker Compose"></a>Docker Compose</h2><p>通常一个Node.js工程都需要数据库，我们将数据库也放在一个容器里面，那个这个Node.js工程所在的容器需要连到数据库所在的容器。我们可通过另一个Dockerfile创建一个数据库容器，并初始化一些数据。<br><br>这是创建数据库容器的Dockerfile<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">FROM mysql:5</span><br><span class="line"></span><br><span class="line">ENV MYSQL_ROOT_PASSWORD 123 </span><br><span class="line">ENV MYSQL_DATABASE users </span><br><span class="line">ENV MYSQL_USER users_service </span><br><span class="line">ENV MYSQL_PASSWORD 123</span><br><span class="line"></span><br><span class="line">ADD setup.sql /docker-entrypoint-initdb.d #任何添加到镜像的 /docker-entrypoint-initdb.d目录的.sql或者.sh文件会在搭建DB的时候执行。</span><br></pre></td></tr></table></figure></p><p>docker build -t test-database . <br><br>docker run –name db test-database <br><br>这样需要通过docker run -it -p 8123:8123 –link db:db -e DATABASE_HOST=DB node4将node4容器连接到数据库容器，才能在Node.js工程中，访问数据库。<br><br>这样看似解决了问题，但如果Node.js工程还依赖Redis、RabbitMQ和MongoDb等，如果是手动来处理这些容器的连接，简直是费时费力，这时候就可以用docker-compose来构建一次构建多个容器间的连接。<br><br>Docker Compose需要一个docker-compose.yml文件，下面是一个示例：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">version: '2' </span><br><span class="line">services: </span><br><span class="line"> users-service:</span><br><span class="line"> build: ./node4</span><br><span class="line"> ports:</span><br><span class="line"> - "8123:8123"</span><br><span class="line"> depends_on:</span><br><span class="line"> - db</span><br><span class="line"> environment:</span><br><span class="line"> - DATABASE_HOST=db</span><br><span class="line"> db:</span><br><span class="line"> build: ./test-database</span><br></pre></td></tr></table></figure></p><p>通过执行<code>docker-compose build</code>指令，就可以构建docker-compose.yml文件里列出的每个镜像，通过<code>docker-compose up</code>将构建的镜像运行起来，<code>docker-compose down</code>停止运行的镜像。<br><br>在docker-compose.yml中，每个servies下的每个服务的build值告诉到哪里找到Dockerfile。<br></p><p><em>参考：<a href="http://blog.jobbole.com/103069/" target="_blank" rel="noopener">http://blog.jobbole.com/103069/</a></em></p>]]></content>
<summary type="html">
<p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/docker.jpg" alt="docker"><br>Docker让开发人员易于快速构建可随时运行的容器化应用程序，它大大简化了管理和部署应用程序的任务。下面主要介绍一些基本指令和用法，由于主要是在Mac上使用，所以有些东西在Mac上才会涉及到。<br>
</summary>
<category term="Docker" scheme="http://liu-xin.me/tags/Docker/"/>
</entry>
<entry>
<title>Effective JavaScript（一）</title>
<link href="http://liu-xin.me/2016/05/14/Effective%20JavaScript%EF%BC%88%E4%B8%80%EF%BC%89/"/>
<id>http://liu-xin.me/2016/05/14/Effective JavaScript（一）/</id>
<published>2016-05-14T12:00:38.000Z</published>
<updated>2018-10-06T16:04:38.283Z</updated>
<content type="html"><![CDATA[<p>最近在读《Effective JavaScript》这本书，书中讲解了很多JS的语言细节和一些最佳实践，有一些是我平时使用时没有注意的，也有一些是自己踩过的坑，下面就列举一些我所留意的知识点。<br><a id="more"></a></p><h2 id="Number"><a href="#Number" class="headerlink" title="Number"></a>Number</h2><p>JS中的数字都是双精度的浮点数。JS中的整数仅仅是双精度浮点数的一个子集，而不是一个单独的数据类型。<br><br>JS的浮点数在运算的时候存在精度陷阱。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">0.1 + 0.2 // 0.30000000000000004</span><br></pre></td></tr></table></figure></p><p>因为浮点数运算只能产生近似的结果，四舍五入到最接近的可表示的实数，当执行一系列的运算，随着舍入误差的积累，运算结果会越来越不精确。<br><br>若想要使<em>0.1+0.2</em>等于0.3，可以使用<em>Number((0.1+0.2).toFixed(1))</em> 来实现，注意toFixed()生成的是字符串，所以必须用Number来转换成数字。</p><p>NaN是JS中的一个特殊数字，它是唯一一个不等于本身的数字。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">var a = Number(b); // a为NaN, 因为b未定义</span><br><span class="line">a == a; // false</span><br><span class="line">a != a; // true</span><br></pre></td></tr></table></figure></p><h2 id="JS中的7个假值"><a href="#JS中的7个假值" class="headerlink" title="JS中的7个假值"></a>JS中的7个假值</h2><p>JS中只有7个布尔类型为false的值，它们分别是<code>false(Boolean)</code>、<code>0(Number)</code>、<code>-0(Nubmer)</code>、<code>""(String)</code>、<code>null(Object)</code>、<code>NaN(Number)</code>、<code>undefined</code>。</p><p><code>注意，空对象{}和空数组[]它们的布尔类型为true。</code></p><h2 id="toString-和String"><a href="#toString-和String" class="headerlink" title="toString()和String()"></a>toString()和String()</h2><p>这两个方法都可以将一个值转换成一个字符串。</p><h4 id="toStirng"><a href="#toStirng" class="headerlink" title="toStirng()"></a>toStirng()</h4><ul><li>大多数值都有toString()方法，null和undefined是没有的。</li><li>对应字符串型的值也可以使用toString()方法，它会返回该字符串的一个副本。</li><li>toString()方法可以传递一个参数，表示数值的基数。（二进制，十进制）<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">var t = 8;</span><br><span class="line">t.toString(2); // '1000', 对于非数字类型设置toString()的参数是无效的。</span><br></pre></td></tr></table></figure></li></ul><h4 id="String"><a href="#String" class="headerlink" title="String()"></a>String()</h4><p>任何值都可以使用String()方法。首先如果值有toString()方法，那么则使用该方法。其次，如果没有toString()方法，那就是null返回’null’，undefined返回’undefined’。</p><h2 id="encodeURI和encodeURIComponent"><a href="#encodeURI和encodeURIComponent" class="headerlink" title="encodeURI和encodeURIComponent"></a>encodeURI和encodeURIComponent</h2><p>encodeURI和encodeURIComponent是把字符编码成UTF-8。<br><br>encodeURI方法不会对ASCII字母、数字、~!@#$%&<em>()=:/,;?+-_这些字符编码。<br><br>encodeURIComponent方法不会对ASCII字母、数字、~!</em>()’这些字符编码。<br><br>所以encodeURIComponent比encodeURI的编码范围更广。<br><br>它们对应的解码方法分别是decodeURI和decodeURIComponent。<br></p><h2 id="闭包"><a href="#闭包" class="headerlink" title="闭包"></a>闭包</h2><p>闭包存储的是外部变量的引用而不是值，并能读写这些值。<br><br>下面这段代码输出什么？<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">function wrapElements(a) {</span><br><span class="line"> var result = [], i, n;</span><br><span class="line"> for (i = 0; len = a.length; i < len; ++i) {</span><br><span class="line"> result[i] = function() { return a[i]; }; // ①</span><br><span class="line"> }</span><br><span class="line"> return result;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">var wrapped = wrapElements([10, 20, 30, 40, 50]);</span><br><span class="line">var f = wrapped[0];</span><br><span class="line">f(); // ?</span><br></pre></td></tr></table></figure></p><p>我刚开始以为是50，但实际是undefined。<br><br>因为闭包存储的是外部变量的引用，所以注释①那行代码中a[i]中的i其实是对i的引用，此时i的值为5，a[5]没有值，所以为undefined。</p>]]></content>
<summary type="html">
<p>最近在读《Effective JavaScript》这本书，书中讲解了很多JS的语言细节和一些最佳实践，有一些是我平时使用时没有注意的，也有一些是自己踩过的坑，下面就列举一些我所留意的知识点。<br>
</summary>
<category term="JavaScript" scheme="http://liu-xin.me/tags/JavaScript/"/>
</entry>
<entry>
<title>在JavaScript中实现AOP</title>
<link href="http://liu-xin.me/2016/01/17/%E5%9C%A8JavaScript%E4%B8%AD%E5%AE%9E%E7%8E%B0AOP/"/>
<id>http://liu-xin.me/2016/01/17/在JavaScript中实现AOP/</id>
<published>2016-01-17T08:57:37.000Z</published>
<updated>2018-10-06T16:04:38.285Z</updated>
<content type="html"><![CDATA[<p>有一个公用的代码，可能在很多地方都会被用到，那么现在要做的就是，需要这个方法跑起来之前走一些东西，在这个方法跑完之后，还在处理一些东西。<br><a id="more"></a></p><h2 id="概念解读"><a href="#概念解读" class="headerlink" title="概念解读"></a>概念解读</h2><p>AOP(Aspect Oriented Programming)，面向切面编程，主要实现的目的是针对业务处理过程中的切面进行提取，它所面对的是处理过程中的某个步骤或阶段，以获得逻辑过程中各部分之间低耦合性的隔离效果。<br>AOP的主要思想是把一些与核心业务无关但又在多个模块使用的功能分离出来，然后动态给业务模块添加上 需要的功能。<br>AOP可以实现对业务无侵入式地干扰。</p><h2 id="代码实战"><a href="#代码实战" class="headerlink" title="代码实战"></a>代码实战</h2><p>如果我们想得到某个函数的执行的开始时间和结束时间，可能会写出如下的代码：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">function test1(){</span><br><span class="line"> console.log(1);</span><br><span class="line">}</span><br><span class="line">console.log('test1 start:', Date.now());</span><br><span class="line">test1();</span><br><span class="line">console.log('test2 start:', Date.now());</span><br></pre></td></tr></table></figure></p><p>上面的代码确实能完成响应的需求，但是扩展性不好，倘若遇到需要得到多个函数它们执行的开始和结束时间，如果还是按上面的那种方式，几乎同样的代码会写很多遍，如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">function test1(){</span><br><span class="line"> console.log(1);</span><br><span class="line">}</span><br><span class="line">function test2(){</span><br><span class="line"> console.log(2);</span><br><span class="line">}</span><br><span class="line">function test3(){</span><br><span class="line"> console.log(3);</span><br><span class="line">}</span><br><span class="line">...</span><br><span class="line">console.log('test1 start:', Date.now());</span><br><span class="line">test1();</span><br><span class="line">console.log('test1 start:', Date.now());</span><br><span class="line"></span><br><span class="line">console.log('test2 start:', Date.now());</span><br><span class="line">test2();</span><br><span class="line">console.log('test2 start:', Date.now());</span><br><span class="line"></span><br><span class="line">console.log('test3 start:', Date.now());</span><br><span class="line">test3();</span><br><span class="line">console.log('test3 start:', Date.now());</span><br><span class="line">...</span><br></pre></td></tr></table></figure></p><p>像这样很不优雅，代码变得极难维护，这时候若实现AOP就能够很轻松的解决类似的问题。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">function test1(){</span><br><span class="line"> console.log(1);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">Function.prototype.before = function(fn){</span><br><span class="line"> var self = this; </span><br><span class="line"> fn(self.name); </span><br><span class="line"> self.apply(this, arguments); </span><br><span class="line">}</span><br><span class="line">Function.prototype.after= function(fn){</span><br><span class="line"> var self = this;</span><br><span class="line"> self.apply(this, arguments); </span><br><span class="line"> fn(self.name);</span><br><span class="line">}</span><br><span class="line">test1.before(printTime);</span><br><span class="line">test1.after(printTime);</span><br><span class="line"></span><br><span class="line">function printTime () {</span><br><span class="line"> console.log(arguments[0], 'end:', Date.now());</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>每个函数都是Function类型的实例，因为Function通过prototype挂载了before方法和after方法，所以作为Function的实例，test1函数就拥有了before和after方法。现在执行test1.before(printTime)和test1.after(printTime)就会自动打印函数执行的开始和结束时间。<br>但是，在上面的代码中发现test1执行了两次，可以用test1函数作为中转，通过执行before后将before的回调和before一起送到after去来解决这个问题。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">var funcName;</span><br><span class="line">function test1(){</span><br><span class="line"> console.log(1);</span><br><span class="line">}</span><br><span class="line">Function.prototype.before = function(fn){</span><br><span class="line"> var self = this;</span><br><span class="line"> funcName = self.name;</span><br><span class="line"> return function(){</span><br><span class="line"> fn.apply(this, arguments); </span><br><span class="line"> self.apply(self, arguments);</span><br><span class="line"> };</span><br><span class="line">};</span><br><span class="line"></span><br><span class="line">Function.prototype.after = function(fn){</span><br><span class="line"> var self = this;</span><br><span class="line"> return function(){</span><br><span class="line"> self.apply(self, arguments);</span><br><span class="line"> fn.apply(this, arguments);</span><br><span class="line"> };</span><br><span class="line">};</span><br><span class="line"></span><br><span class="line">test1.before(printTime).after(printTime)(); </span><br><span class="line"></span><br><span class="line">function printTime () {</span><br><span class="line"> console.log(funcName, 'end:', Date.now());</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>在上面的代码中，test1函数调用before方法后得到一个匿名函数，该匿名函数继续调用其拥有的after方法，又得到一个匿名函数，最后执行这个匿名函数。最后这个匿名函数做的工作有打印test1()开始执行前的时间，执行test1函数，打印test1()执行后的时间。<br>这样就通过before中的回调作为传递，test1就只执行一次。<br>像最开始的需求那样，若有多个函数需要做相同的事，只需像下面这样就能做到。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">test1.before(printTime).after(printTime)();</span><br><span class="line">test2.before(printTime).after(printTime)();</span><br><span class="line">test3.before(printTime).after(printTime)();</span><br><span class="line">.</span><br><span class="line">.</span><br><span class="line">.</span><br><span class="line">testN.before(printTime).after(printTime)();</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<p>有一个公用的代码，可能在很多地方都会被用到，那么现在要做的就是，需要这个方法跑起来之前走一些东西，在这个方法跑完之后，还在处理一些东西。<br>
</summary>
<category term="JavaScript" scheme="http://liu-xin.me/tags/JavaScript/"/>
<category term="AOP" scheme="http://liu-xin.me/tags/AOP/"/>
</entry>
<entry>
<title>MongoDB中的mapReduce</title>
<link href="http://liu-xin.me/2015/12/21/MongoDB%E4%B8%AD%E7%9A%84mapReduce/"/>
<id>http://liu-xin.me/2015/12/21/MongoDB中的mapReduce/</id>
<published>2015-12-21T07:46:09.000Z</published>
<updated>2018-10-06T16:04:38.283Z</updated>
<content type="html"><![CDATA[<p>MongoDB中的mapReduce一般用于处理大数据集。<br><a id="more"></a></p><h2 id="基本用法"><a href="#基本用法" class="headerlink" title="基本用法"></a>基本用法</h2><p>以下是基本的mapReduce命令的语法：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">>db.collection.mapReduce(</span><br><span class="line"> function() {emit(key,value);}, //map function</span><br><span class="line"> function(key,values) {return reduceFunction}, //reduce function</span><br><span class="line"> {</span><br><span class="line"> out: collection,</span><br><span class="line"> query: document,</span><br><span class="line"> sort: document,</span><br><span class="line"> limit: number</span><br><span class="line"> }</span><br><span class="line">)</span><br></pre></td></tr></table></figure></p><p>使用mapReduce要实现连个函数Map函数和Reduce函数。<br>Map函数调用emit(key, value)，遍历collection中所有的记录，表示集合会按照指定的key进行映射分组，类似group by，分组的结果为value。<br>Reduce函数将对Map函数传递的key与value进行处理。<br>参数说明：</p><ul><li><strong>map:</strong> 映射函数（生成键值对序列，作为reduce函数参数）。</li><li><strong>reduce:</strong> 统计函数，reduce函数的任务就是将key-values编程key-value，也就是把values数组变成一个单一的值value。</li><li><strong>out:</strong> 统计结果存放集合，不指定则使用临时集合，在客户端断开后自动删除。</li><li><strong>query:</strong> 一个筛选条件，只有满足条件的文档才会调用Map函数。（query，limit，sort可随意组合）</li><li><strong>sort:</strong> 排序参数，也就是在发往Map函数前给集合排序，可以优化分组机制。</li><li><strong>limit:</strong> 发往Map函数的文档数量的上限，要是没有limit，单独使用sort的用处不大。</li></ul><h2 id="演示"><a href="#演示" class="headerlink" title="演示"></a>演示</h2><p>插入数据：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">> db.books.find()</span><br><span class="line">{ "_id" : ObjectId("533ee1e8634249165a819cd0"), "name" : "apue", "pagenum" : 1023 }</span><br><span class="line">{ "_id" : ObjectId("533ee273634249165a819cd1"), "name" : "clrs", "pagenum" : 2000 }</span><br><span class="line">{ "_id" : ObjectId("533ee2ab634249165a819cd2"), "name" : "python book", "pagenum" : 600 }</span><br><span class="line">{ "_id" : ObjectId("533ee2b7634249165a819cd3"), "name" : "golang book", "pagenum" : 400 }</span><br><span class="line">{ "_id" : ObjectId("533ee2ca634249165a819cd4"), "name" : "linux book", "pagenum" : 1500 }</span><br></pre></td></tr></table></figure></p><p>Map函数：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">var map = function(){</span><br><span class="line"> var category;</span><br><span class="line"> if ( this.pageNum >= 1000 ){</span><br><span class="line"> category = 'Big Books';</span><br><span class="line"> }else{</span><br><span class="line"> category = 'Small Books';</span><br><span class="line"> }</span><br><span class="line"> emit(category, {name: this.name});</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>Map函数里面会调用emit(key, value)，集合会按照指定的key进行映射分组, 类似group by。this指向每一条被迭代的数据。</p><p>上面执行Map函数后的结果为: (按照category分组, 分组结果是{name: this.name}的list)<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">{"big books",[{name: "apue"}, {name : "linux book"}, {name : "clrs"}]]);</span><br><span class="line">{"small books",[{name: "python book"}, {name : "golang book"}]);</span><br></pre></td></tr></table></figure></p><p>Reduce函数：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">var reduce = function(key, values) {</span><br><span class="line"> var sum = 0;</span><br><span class="line"> values.forEach(function(doc) {</span><br><span class="line"> sum += 1;</span><br><span class="line"> });</span><br><span class="line"> return {books: sum};</span><br><span class="line">};</span><br></pre></td></tr></table></figure></p><p>reduce函数会对Map分组后的数据进行分组简化，注意：在reduce(key,value)中的key就是emit中的key，value为emit分组后的emit(value)的集合, 这里是{name: this.name}的list。</p><p>mapReduce函数：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">> var count = db.books.mapReduce(map, reduce, {out: "book_results"});</span><br><span class="line">> db[count.result].find()</span><br><span class="line">{ "_id" : "big books", "value" : { "books" : 3 } }</span><br><span class="line">{ "_id" : "small books", "value" : { "books" : 2 } }</span><br></pre></td></tr></table></figure></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">> db.books.mapReduce(map, reduce, {out: "book_results"});</span><br><span class="line">{</span><br><span class="line"> "result" : "book_results",</span><br><span class="line"> "timeMillis" : 107,</span><br><span class="line"> "counts" : {</span><br><span class="line"> "input" : 5,</span><br><span class="line"> "emit" : 5,</span><br><span class="line"> "reduce" : 2,</span><br><span class="line"> "output" : 2</span><br><span class="line"> },</span><br><span class="line"> "ok" : 1,</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>具体参数说明：</p><ul><li><strong>result:</strong> 存储结果的collection的名字，这是个临时集合，mapReduce的连接关闭后制动就被删除了。</li><li><strong>timeMillis:</strong> 执行花费的时间，毫秒为单位。</li><li><strong>input:</strong> 满足条件被发送到Map函数的个数。</li><li><strong>emit:</strong> 在Map函数中emit被调用的次数，也就是所有集合中的数据总量。</li><li><strong>output:</strong> 结果集合中的个数。</li></ul><p>查询结果集中的数据：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">> db.book_results.find()</span><br><span class="line">{ "_id" : "big books", "value" : { "books" : 3 } }</span><br><span class="line">{ "_id" : "small books", "value" : { "books" : 2 } }</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<p>MongoDB中的mapReduce一般用于处理大数据集。<br>
</summary>
<category term="数据库" scheme="http://liu-xin.me/tags/%E6%95%B0%E6%8D%AE%E5%BA%93/"/>
<category term="NoSQL" scheme="http://liu-xin.me/tags/NoSQL/"/>
<category term="MongoDB" scheme="http://liu-xin.me/tags/MongoDB/"/>
</entry>
<entry>
<title>yield和yield*</title>
<link href="http://liu-xin.me/2015/12/18/yield%E5%92%8Cyield/"/>
<id>http://liu-xin.me/2015/12/18/yield和yield/</id>
<published>2015-12-18T08:29:52.000Z</published>
<updated>2018-10-06T16:04:38.284Z</updated>
<content type="html"><![CDATA[<p>在ES6的generator函数中经常能见到yield和yield*，开始不是很清楚它们之间有什么区别，在参考了一些资料过后渐渐有一些清楚了。<br><a id="more"></a></p><h2 id="Array与String"><a href="#Array与String" class="headerlink" title="Array与String"></a>Array与String</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">function* GenFunc(){</span><br><span class="line"> yield [1, 2];</span><br><span class="line"> yield* [3, 4];</span><br><span class="line"> yield "56";</span><br><span class="line"> yield* "78";</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">var gen = GenFunc();</span><br><span class="line">console.log(gen.next().value); //[1, 2]</span><br><span class="line">console.log(gen.next().value); //3</span><br><span class="line">console.log(gen.next().value); //4</span><br><span class="line">console.log(gen.next().value); //"56"</span><br><span class="line">console.log(gen.next().value); //7</span><br><span class="line">console.log(gen.next().value); //8</span><br></pre></td></tr></table></figure><p>从上面的代码可以看出，yield*后面如果跟的是一个Array或String，每执行一次next()函数，将会迭代Array或String中的一个元素，而如果是普通的yield则直接返回这个对象。</p><h2 id="arguments"><a href="#arguments" class="headerlink" title="arguments"></a>arguments</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">function* GenFunc(){</span><br><span class="line"> yield arguments;</span><br><span class="line"> yield* arguments;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">var gen = GenFunc(1, 2);</span><br><span class="line">console.log(gen.next().value); //{ '0': 1, '1': 2}</span><br><span class="line">console.log(gen.next().value); //1</span><br><span class="line">console.log(gen.next().value); //2</span><br></pre></td></tr></table></figure><p>arguments的情况和Array或String情况一样，说明yield*后面如果跟的是一个可迭代对象，每执行一次next()函数，将会迭代一次这个对象。</p><h2 id="Generator"><a href="#Generator" class="headerlink" title="Generator"></a>Generator</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">function* Gen1(){</span><br><span class="line"> yield 2;</span><br><span class="line"> yield 3;</span><br><span class="line"> return 'liuxin';</span><br><span class="line">}</span><br><span class="line">function* Gen2(){</span><br><span class="line"> yield 1;</span><br><span class="line"> var a = yield* Gen1();</span><br><span class="line"> console.log(a); //打印'liuxin'</span><br><span class="line"> yield 4;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">var gen2 = Gen2();</span><br><span class="line">console.log(gen2.next().value); //1</span><br><span class="line">console.log(gen2.next().value); //2</span><br><span class="line">console.log(gen2.next().value); //3</span><br><span class="line">console.log(gen2.next().value); //4</span><br></pre></td></tr></table></figure><p>yield<em>后面如果跟的是一个Generator函数，那么便会执行这个Generator函数，同时yield</em>这个表达式的值就是这个Generator函数的返回值。</p><h2 id="Object"><a href="#Object" class="headerlink" title="Object"></a>Object</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">function* GenFunc(){</span><br><span class="line"> yield {a: '1', b: '2'};</span><br><span class="line"> yield* {a: '1', b: '2'};</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">var gen = GenFunc();</span><br><span class="line">console.log(gen.next().value); //{ a: '1', b: '2'}</span><br><span class="line">console.log(gen.next().value); //1</span><br><span class="line">console.log(gen.next().value); //2</span><br></pre></td></tr></table></figure><p>情况和arguments的情况一样。</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>根据上面的所有代码可以知道，yield<em>后面接受一个iterable object作为参数，然后去迭代(iterate)这个迭代器（iterable object)，同时yield</em>本身这个表达式的值就是迭代器迭代完成时的返回值，以及yield*可以用来在一个generator函数里“执行”另一个generator函数，并可取得其返回值。</p><p>参考：<a href="http://taobaofed.org/blog/2015/11/19/yield-and-delegating-yield/" target="_blank" rel="noopener">http://taobaofed.org/blog/2015/11/19/yield-and-delegating-yield/</a></p>]]></content>
<summary type="html">
<p>在ES6的generator函数中经常能见到yield和yield*，开始不是很清楚它们之间有什么区别，在参考了一些资料过后渐渐有一些清楚了。<br>
</summary>
<category term="JavaScript" scheme="http://liu-xin.me/tags/JavaScript/"/>
<category term="Node.js" scheme="http://liu-xin.me/tags/Node-js/"/>
</entry>
<entry>
<title>SQL成长记</title>
<link href="http://liu-xin.me/2015/11/19/SQL%E6%88%90%E9%95%BF%E8%AE%B0/"/>
<id>http://liu-xin.me/2015/11/19/SQL成长记/</id>
<published>2015-11-19T02:45:40.000Z</published>
<updated>2018-10-06T16:04:38.283Z</updated>
<content type="html"><![CDATA[<p>这一段时间在进行服务器端的开发，在开发的过程需要遍写大量的SQL语句来操作数据库，对于之前SQL还是菜鸟水平的我来说在开发的过程中遇到很多之前不太熟悉的地方，有些业务需要的SQL较为复杂，需要连接好几张表，数据呈现的要求也多种多样，最后也还算顺利的完成了业务。<br><a id="more"></a><br>下面是一些在这个过程中让我对SQL有更深入认识的SQL语句：</p><h2 id="模糊查询并按匹配度进行排序"><a href="#模糊查询并按匹配度进行排序" class="headerlink" title="模糊查询并按匹配度进行排序"></a>模糊查询并按匹配度进行排序</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select name from user where name like '%foo%' order by abs(length(name)-length('foo'));</span><br></pre></td></tr></table></figure><p>这是一种按匹配度排序很简陋的做法，在很多情况下都存在问题，如果是对查询的结果按匹配度排序有着重度的需求，最好不要使用这个。</p><h2 id="分组统计数量"><a href="#分组统计数量" class="headerlink" title="分组统计数量"></a>分组统计数量</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select count(1) 'count',class from user where gender='male' group by class;</span><br></pre></td></tr></table></figure><p>上面的SQL语句是在user表中查询每个班男生的数量，通过group by class做到了按班分组，通过count(1)做到了统计数量。</p><h2 id="随机查询指定数量的数据"><a href="#随机查询指定数量的数据" class="headerlink" title="随机查询指定数量的数据"></a>随机查询指定数量的数据</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select id,name from user where type=5 order by rand() limit 10;</span><br></pre></td></tr></table></figure><p>在这里用到了SQL的内置函数rand()用于获取随机数据，不过处于性能考虑，在SQL语句中应尽量少用内置函数。</p><h2 id="在查询字段中指定一个常量"><a href="#在查询字段中指定一个常量" class="headerlink" title="在查询字段中指定一个常量"></a>在查询字段中指定一个常量</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select k.* from (select id,create_date,content,'homework' as 'type' from comments where isHomework=1 union select id,create_date,content,'post' as 'type' from posts) k where k.account_id='123456' order by k.create_date desc;</span><br></pre></td></tr></table></figure><p>上面的SQL语句在查询comments表和posts表是分别在查询字段中增加了常量homework和post，并指定了列名为type，也就是在临时表k中，存在名为type的列，它下面的值要么是homework，要么是post。</p>]]></content>
<summary type="html">
<p>这一段时间在进行服务器端的开发，在开发的过程需要遍写大量的SQL语句来操作数据库，对于之前SQL还是菜鸟水平的我来说在开发的过程中遇到很多之前不太熟悉的地方，有些业务需要的SQL较为复杂，需要连接好几张表，数据呈现的要求也多种多样，最后也还算顺利的完成了业务。<br>
</summary>
<category term="数据库" scheme="http://liu-xin.me/tags/%E6%95%B0%E6%8D%AE%E5%BA%93/"/>
<category term="SQL" scheme="http://liu-xin.me/tags/SQL/"/>
</entry>
<entry>
<title>在Redis中进行分页排序查询</title>
<link href="http://liu-xin.me/2015/11/17/%E5%9C%A8Redis%E4%B8%AD%E8%BF%9B%E8%A1%8C%E5%88%86%E9%A1%B5%E6%8E%92%E5%BA%8F%E6%9F%A5%E8%AF%A2/"/>
<id>http://liu-xin.me/2015/11/17/在Redis中进行分页排序查询/</id>
<published>2015-11-17T07:35:01.000Z</published>
<updated>2021-01-08T15:08:30.200Z</updated>
<content type="html"><![CDATA[<p>Redis是一个高效的内存数据库，它支持包括String、List、Set、SortedSet和Hash等数据类型的存储，在Redis中通常根据数据的key查询其value值，Redis没有条件查询，在面对一些需要分页或排序的场景时（如评论，时间线），Redis就不太好不处理了。<br><a id="more"></a><br>前段时间在项目中需要将每个主题下的用户的评论组装好写入Redis中，每个主题会有一个topicId，每一条评论会和topicId关联起来，得到大致的数据模型如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> topicId: 'xxxxxxxx',</span><br><span class="line"> comments: [</span><br><span class="line"> {</span><br><span class="line"> username: 'niuniu',</span><br><span class="line"> createDate: 1447747334791,</span><br><span class="line"> content: '在Redis中分页',</span><br><span class="line"> commentId: 'xxxxxxx',</span><br><span class="line"> reply: [</span><br><span class="line"> {</span><br><span class="line"> content: 'yyyyyy'</span><br><span class="line"> username: 'niuniu'</span><br><span class="line"> },</span><br><span class="line"> ...</span><br><span class="line"> ]</span><br><span class="line"> },</span><br><span class="line"> ...</span><br><span class="line"> ]</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>将评论数据从MySQL查询出来组装好存到Redis后，以后每次就可以从Redis获取组装好的评论数据，从上面的数据模型可以看出数据都是key-value型数据，无疑要采用hash进行存储，但是每次拿取评论数据时需要分页而且还要按createDate字段进行排序，hash肯定是不能做到分页和排序的。</p><p>那么，就挨个看一下Redis所支持的数据类型：</p><blockquote><p><strong>String: </strong>主要用于存储字符串，显然不支持分页和排序。<br><strong>Hash: </strong>主要用于存储key-value型数据，评论模型中全是key-value型数据，所以在这里Hash无疑会用到。<br><strong>List: </strong>主要用于存储一个列表，列表中的每一个元素按元素的插入时的顺序进行保存，如果我们将评论模型按createDate排好序后再插入List中，似乎就能做到排序了，而且再利用List中的LRANGE key start stop指令还能做到分页。嗯，到这里List似乎满足了我们分页和排序的要求，但是评论还会被删除，就需要更新Redis中的数据，如果每次删除评论后都将Redis中的数据全部重新写入一次，显然不够优雅，效率也会大打折扣，如果能删除指定的数据无疑会更好，而List中涉及到删除数据的就只有LPOP和RPOP这两条指令，但LPOP和RPOP只能删除列表头和列表尾的数据，不能删除指定位置的数据，所以List也不太适合。<br><strong>Set: </strong>主要存储无序集合，无序！排除。<br><strong>SortedSet: </strong>主要存储有序集合，SortedSet的添加元素指令<em>ZADD key score member [[score,member]…]</em>会给每个添加的元素member绑定一个用于排序的值score，SortedSet就会根据score值的大小对元素进行排序，在这里就可以将createDate当作score用于排序，SortedSet中的指令<em>ZREVRANGE key start stop</em>又可以返回指定区间内的成员，可以用来做分页，SortedSet的指令ZREM key member可以根据key移除指定的成员，能满足删评论的要求，所以，SortedSet在这里是最适合的。</p></blockquote><p>所以，我需要用到的数据类型有SortSet和Hash，SortSet用于做分页排序，Hash用于存储具体的键值对数据，我画出了如下的结构图：</p><p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/redis.png" alt="结构图"></p><p>在上图的SortSet结构中将每个主题的topicId作为set的key，将与该主题关联的评论的createDate和commentId分别作为set的score和member，commentId的顺序就根据createDate的大小进行排列。<br>当需要查询某个主题某一页的评论时，就可主题的topicId通过指令<em>zrevrange topicId (page-1)×10 (page-1)×10+perPage</em>这样就能找出某个主题下某一页的按时间排好顺序的所有评论的commintId。page为查询第几页的页码，perPage为每页显示的条数。<br>当找到所有评论的commentId后，就可以把这些commentId作为key去Hash结构中去查询该条评论对应的内容。<br>这样就利用SortSet和Hash两种结构在Redis中达到了分页和排序的目的。</p>]]></content>
<summary type="html">
<p>Redis是一个高效的内存数据库，它支持包括String、List、Set、SortedSet和Hash等数据类型的存储，在Redis中通常根据数据的key查询其value值，Redis没有条件查询，在面对一些需要分页或排序的场景时（如评论，时间线），Redis就不太好不处理了。<br>
</summary>
<category term="数据库" scheme="http://liu-xin.me/tags/%E6%95%B0%E6%8D%AE%E5%BA%93/"/>
<category term="NoSQL" scheme="http://liu-xin.me/tags/NoSQL/"/>
<category term="Redis" scheme="http://liu-xin.me/tags/Redis/"/>
</entry>
<entry>
<title>JavaScript中的this</title>
<link href="http://liu-xin.me/2015/11/16/JavaScript%E4%B8%AD%E7%9A%84this/"/>
<id>http://liu-xin.me/2015/11/16/JavaScript中的this/</id>
<published>2015-11-16T02:17:01.000Z</published>
<updated>2021-01-08T15:01:51.059Z</updated>
<content type="html"><![CDATA[<p>JavaScript中的this在运行期进行绑定，这使得JavaScript中的this关键字具备多重含义。<br><a id="more"></a><br>JavaScript中的this到底指向什么？可以用下面一张图来解释：</p><p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/this.jpg" alt="this的指向"></p><h2 id="指向全局对象的例子"><a href="#指向全局对象的例子" class="headerlink" title="指向全局对象的例子"></a>指向全局对象的例子</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">var point = {</span><br><span class="line">x: 0,</span><br><span class="line">y: 0,</span><br><span class="line">moveTo: function(x, y){</span><br><span class="line">//内部函数</span><br><span class="line">var moveX = function(x){</span><br><span class="line">this.x = x; //this指向什么？window或global</span><br><span class="line">}；</span><br><span class="line">//内部函数</span><br><span class="line">var moveY = function(y){</span><br><span class="line">this.y = y; //this指向什么？window或global</span><br><span class="line">}；</span><br><span class="line">moveX(x);</span><br><span class="line">moveY(y);</span><br><span class="line">}</span><br><span class="line">};</span><br><span class="line">point.moveTo(1, 1);</span><br><span class="line">point.x; //=>0</span><br><span class="line">point.y; //=>0</span><br><span class="line">x; //=>1</span><br><span class="line">y; //=>1</span><br></pre></td></tr></table></figure><p>在上面的代码中，moveX(x)函数调用既不是用new进行调用，也不是用dot(.)进行调用，所以this指向window。</p><h2 id="构造函数的例子"><a href="#构造函数的例子" class="headerlink" title="构造函数的例子"></a>构造函数的例子</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">function Point(x, y){</span><br><span class="line">this.x = x; // this?</span><br><span class="line">this.y = y; // this?</span><br><span class="line">}</span><br><span class="line">var np = new Point(1, 1);</span><br><span class="line">np.x; //1</span><br><span class="line">var p = Point(2, 2);</span><br><span class="line">p.x; //error, p是一个空对象undefined</span><br><span class="line">window.x; //2</span><br></pre></td></tr></table></figure><p>上面的代码中var np = new Point(1, 1)是用new进行调用，所以this指向创建的对象np，所以np.x就为1。<br>var p = Point(2, 2)函数调用既不是用new进行调用，也不是用dot(.)进行调用，所以this指向window。 </p><h2 id="call和apply进行调用的例子"><a href="#call和apply进行调用的例子" class="headerlink" title="call和apply进行调用的例子"></a>call和apply进行调用的例子</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">function Point(x, y){</span><br><span class="line">this.x = x;</span><br><span class="line">this.y = y;</span><br><span class="line">this.moveTo = function(x, y){</span><br><span class="line">this.x = x;</span><br><span class="line">this.y = y;</span><br><span class="line">}；</span><br><span class="line">}</span><br><span class="line">var p1 = new Point(0, 0);</span><br><span class="line">var p2 = {x: 0, y: 0};</span><br><span class="line">p1.moveTo.apply(p2, [10, 10]); //apply实际上为p2.moveTo(10, 10)</span><br><span class="line">p2.x //10</span><br></pre></td></tr></table></figure><p>apply和call这两个方法可以改变函数执行的上下文，即改变this绑定的对象。p1.moveTo.apply(p2, [10,10])实际上是p2.moveTo(10, 10)。那么p2.moveTo(10, 10)可解释为不是new调用也而是dot(.)调用，所以this绑定的对象是p2。</p>]]></content>
<summary type="html">
<p>JavaScript中的this在运行期进行绑定，这使得JavaScript中的this关键字具备多重含义。<br>
</summary>
<category term="JavaScript" scheme="http://liu-xin.me/tags/JavaScript/"/>
<category term="this" scheme="http://liu-xin.me/tags/this/"/>
</entry>
<entry>
<title> 这个博客怎么搭建的？</title>
<link href="http://liu-xin.me/2015/11/15/%E8%BF%99%E4%B8%AA%E5%8D%9A%E5%AE%A2%E6%80%8E%E4%B9%88%E6%90%AD%E5%BB%BA%E7%9A%84%EF%BC%9F/"/>
<id>http://liu-xin.me/2015/11/15/这个博客怎么搭建的？/</id>
<published>2015-11-15T08:24:31.000Z</published>
<updated>2021-01-08T15:04:59.396Z</updated>
<content type="html"><![CDATA[<p>本站是一个静态的个人博客网站，托管在 <a href="https://pages.github.com" target="_blank" rel="noopener">Github Pages</a> 上面。<br><a id="more"></a></p><h2 id="Github-Pages"><a href="#Github-Pages" class="headerlink" title="Github Pages"></a>Github Pages</h2><p>要想使用 Github Pages 服务，则需要在 <a href="https://github.com" target="_blank" rel="noopener">Github</a> 上创建一个与自己用户名有关的仓库（当然，得先有 Github 账号），如用户名若为 niuniu，则创建的仓库名称应为 niuniu.github.io。</p><h2 id="GoDaddy"><a href="#GoDaddy" class="headerlink" title="GoDaddy"></a>GoDaddy</h2><p>liu-xin.me 这个域名是在 <a href="https://www.godaddy.com/" target="_blank" rel="noopener">GoDaddy</a> 上申请的。申请完后如果想要通过自己的域名访问在Github上创建的仓库，则需要配置一下域名的 DNS 解析，DNS 解析所需要的 IP 地址可以在<a href="https://help.github.com/en/articles/setting-up-an-apex-domain#configuring-a-records-with-your-dns-provider" target="_blank" rel="noopener">这里</a>找到，可以选择其中任意一个 IP 作为解析地址，或者通过命令 <code>dig 仓库名</code>也可以拿到要解析的 IP，比如 <code>dig niuniu.github.io</code>。我的配置如下图：</p><p><img src="https://raw.githubusercontent.com/beyond5959/images/main/uPic/img/godaddyDNS.png" alt="域名的DNS配置"></p><p>当然到这里还不够，还需要在 Github 仓库里创建一个名为 CNAME 的文件，CNAME 文件里的内容为申请的域名，我的 CNAME 文件里的内容就为 liu-xin.me。</p><p>除此之外，仓库里面还需要一个 index.html 文件，在 index.html 中通过 HTML 编辑好自己想要的内容后，然后访问自己的域名就可以看到自己编辑的东西了。</p><h2 id="Hexo"><a href="#Hexo" class="headerlink" title="Hexo"></a>Hexo</h2><p>本站采用 <a href="https://hexo.io" target="_blank" rel="noopener">Hexo</a>，这里不对 Hexo 作详细的介绍，需要注意的是当把自己的博客部署到 Github 上时，需要把 CNAME 文件放到通过 Hexo 生成的工程的 source 目录下，如果需要 README.md 文件也需要将该文件放在 source 目录下，并在工程的 _config.yml 文件中增加 <code>skip_render: README.md</code> 的配置。</p>]]></content>
<summary type="html">
<p>本站是一个静态的个人博客网站，托管在 <a href="https://pages.github.com" target="_blank" rel="noopener">Github Pages</a> 上面。<br>
</summary>
<category term="Github Pages" scheme="http://liu-xin.me/tags/Github-Pages/"/>
<category term="GoDaddy" scheme="http://liu-xin.me/tags/GoDaddy/"/>
<category term="Hexo" scheme="http://liu-xin.me/tags/Hexo/"/>
</entry>
</feed>