网卡配置惊魂记

这是我近期遇到的一个集群的问题,涉及到的主要是硬件的配置导致的软件层问题。起初发现这个问题是由于我们开发的基于YARN的实时流系统某些机器的tps非常低,一开始并没有想到是硬件的问题,所以一直在排查实时框架的问题。最后发现连ping值都很低,才想到是OS配置相关的问题。
这是ping的延迟:

64 bytes from 10.39.****: icmp_seq=34 ttl=63 time=27.4 ms
64 bytes from 10.39.****: icmp_seq=35 ttl=63 time=21.2 ms
64 bytes from 10.39.****: icmp_seq=36 ttl=63 time=5.95 ms
64 bytes from 10.39.****: icmp_seq=37 ttl=63 time=16.5 ms
64 bytes from 10.39.****: icmp_seq=38 ttl=63 time=12.3 ms

底下是正常的ping的延迟

64 bytes from 10.39.****: icmp_seq=13 ttl=63 time=0.230 ms
64 bytes from 10.39.****: icmp_seq=14 ttl=63 time=0.203 ms
64 bytes from 10.39.****: icmp_seq=15 ttl=63 time=0.255 ms
64 bytes from 10.39.****: icmp_seq=16 ttl=63 time=0.229 ms

推测就是网卡的问题,我们是两个千兆网卡bonding,查看/proc/interrupt发现每个网卡只绑到了一个core上,这个core一直跑满,并且丢包,正常的网卡配置会使用多队列,将不同队列亲和到不同的core上,提高tps等。相关的资料可以参见这篇文章:网卡多队列简介.
由于实时系统发送的大多是小包,所以才会有这么大的影响,解决方法比较坑爹,配置好多队列后需要重启服务器。

Some Note About Deploy Hadoop Clusters

1.Download source code, using maven to compile and package, remember to compile the native for the same OS version.

mvn package -Pdist -Pnative -Dtar -DskipTests

2.Edit the core-site.xml, hdfs-site.xml, mapred-site.xml and yarn-site.xml file
you should add some JVM parameters or log position in /etc/bashrc, just a example below:

export JAVA_HOME=/usr/local/jdk1.7.0_67
export JRE_HOME=/usr/local/jdk1.7.0_67/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export HADOOP_HOME=/usr/local/hadoop-2.4.0
export HADOOP_LOG_DIR=/data0/hadoop/log/hadoop
export HADOOP_PID_DIR=/data0/hadoop/pid/hadoop
export YARN_LOG_DIR=/data0/hadoop/log/yarn
export YARN_PID_DIR=/data0/hadoop/pid/yarn
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_NAMENODE_OPTS=" -Xmx20480m -Xms20480m -Xmn3072m -verbose:gc -Xloggc:/data0/hadoop/gclog/namenode.gc.log -XX:ErrorFile=/data0/hadoop/gclog/hs_err_pid.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=85 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=1000 -XX:+CMSClassUnloadingEnabled -XX:+DisableExplicitGC -Dcom.sun.management.jmxremote.port=6000 -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/usr/local/hadoop-2.4.0/etc/hadoop/jmxremote.password"
export YARN_RESOURCEMANAGER_OPTS=" -Xmx10240m -Xms10240m -Xmn3072m -verbose:gc -Xloggc:/data0/hadoop/gclog/yarn.gc.log -XX:ErrorFile=/data0/hadoop/gclog/hs_err_pid.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCMSCompactAtFullCollection -XX:CMSMaxAbortablePrecleanTime=1000 -XX:+CMSClassUnloadingEnabled -XX:+DisableExplicitGC -Dcom.sun.management.jmxremote.port=6001 -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/usr/local/hadoop-2.4.0/etc/hadoop/jmxremote.password"
ulimit -u 65535

3.Put the packaged code into all the servers, including namenodes, resourcemanagers, nodemanagers and datanodes
4.startup journalnode first, i assume you use qjournal

hadoop-daemon.sh start journalnode

5.format namenode for specific namespace

hdfs namenode -format
if you use federation, make sure the cluster id are the same,so if the first ns cluster id is abcdefg, the second ns should format with the cluster id, hdfs namenode -format -clusterId=abcdefg

6.init the standby namenode for the same namespace

hdfs namenode -bootstrapStandby

7.start namenodes and datanodes

hadoop-daemon.sh start namenode
hadoop-daemon.sh start datanode

8.transition to active namenode

for example namespace is ns, active namenode is nn1
 hdfs haadmin -ns ns -transitionToActive nn1

9.mkdir dir for hadoop user and mapred(the user to startup resource manager and history server) user
10.mkdir for history server
for example the mapred-site.xml set history directory

mapreduce.jobhistory.intermediate-done-dir
hdfs://hbasens/hadoop/history/tmp
true
mapreduce.jobhistory.done-dir
hdfs://hbasens/hadoop/history/done
true

you have to set directory like this

hdfs dfs -mkdir -p /hadoop/history/tmp
hdfs dfs -chown -R mapred:mapred /hadoop/history
hdfs dfs -chmod -R 1777 /hadoop/history/tmp
hdfs dfs -mkdir -p /hadoop/history/done
hdfs dfs -chmod -R 1777 /hadoop/history/done

11.startup resourcemanager and nodemanager and mr history server

yarn-daemon.sh start resourcemanager
yarn-daemon.sh start nodemanager
mr-jobhistory-daemon.sh start historyserver

使用grunt搭建前后端分离的开发环境

由于OPOA的兴起,前后端分离的开发模式已经成为web开发的趋势,搭建一个能够快速进行并行开发独立联调的环境是非常重要的,感谢grunt gulp的前端构建工具,让这个目标变为了可能。下面就介绍一种grunt配置方式能够实现如下目标,让f2er能够拷贝即用

  • 1 搭建server
  • 2 监听js html css文件变化自动刷新(不使用插件)
  • 3 实现代理功能,访问后端的api接口 调试前端src下的文件。

假设我们的web站点为 apis路径下是后台的接口,src下是所有的前端文件 index.html是入口的文件,package.json和Gruntfile文件如下:

{
  "name": "webtools",
  "version": "0.0.1",
  "description": "grunt tools for web development",
  "main": "Gruntfile.js",
  "repository": {
    "type": "git",
    "url": ""
  },
  "author": "zhangmeng<zhangmeng712@126.com>",
  "license": "ISC",
  "devDependencies": {
    "grunt": "^0.4.5",
    "grunt-contrib-connect": "^1.0.0",
    "grunt-contrib-watch": "^0.6.1",
    "grunt-connect-proxy": "",
    "connect-livereload": "^0.5.4",
    "serve-static": "^1.10.2"
  }
}

var HOSTNAME = '0.0.0.0'; var LIVERELOAD_PORT = 9991; var SERVER_PORT = 9966; var serveStatic = require('serve-static'); //如果请求header中有验证信息如cookie则需要自己配置 var cookie = 'xxxxxxx'; module.exports = function (grunt) { require('load-grunt-tasks')(grunt); grunt.initConfig({ connect: { options: { port: SERVER_PORT, hostname: HOSTNAME }, proxies: [ { context: '/apis', // 这是你希望出现在grunt serve服务中的路径,比如这里配置的是http://127.0.0.1:9000/api/ host: 'dev.alp.xx-inc.com', port: 80, // 远端服务器端口 headers: { 'cookie': cookie, 'host': 'dev.alp.xx-inc.com' }, rewrite: { '^/apis/': '/' //地址映射策略,从context开始算,把前后地址做正则替换,如果远端路径和context相同则不用配置。 } } ], livereload: { options: { open: { target: 'http://' + HOSTNAME + ':' + SERVER_PORT + '/index.html?debug&mock&local' }, // 通过LiveReload脚本,让页面重新加载。 middleware: function (connect, options) { var proxyMid = require('grunt-connect-proxy/lib/utils').proxyRequest; var livereloadMid = require('connect-livereload')({port: LIVERELOAD_PORT}); var serveMid = serveStatic(__dirname + '/'); var midArr; //判断联调还是mock开发 var isLiantiao = grunt.option('liantiao'); if (isLiantiao) { midArr= [livereloadMid, proxyMid, serveMid ]} else {midArr= [livereloadMid, serveMid ]} return midArr; } } } }, watch: { doc: { files: ['src/**/*.js', 'src/**/*.html', 'src/**/*.css'], tasks: [] }, options: { livereload: LIVERELOAD_PORT, spawn: true } } }); grunt.registerTask('server', ['configureProxies:server', 'connect:livereload', 'watch']); };
    grunt server:livereload #mock调试
    grunt server:livereload --liantiao #前后端联调

知识点

  • load-grunt-tasks 可以帮助我们把package.json中含有grunt-*的内容load下来,代替grunt.loadNpmTasks(‘grunt-xxx’)
  • var serveStatic = require(‘serve-static’); 老的connect版本中使用的是connect.static现在已经都被serve-static中间件代替
  • 如果后端是个通用的接口则可以直接使用proxies的配置方法,如果有些系统需要cookie信息(ssl证书产生)则需要先访问网站,得到对应信息,然后在手动拷贝到Gruntfile.js中去。或者命令行访问 curl -i -L ‘http://dev.alp.xxxx-inc.com/’ -k –cert cer.pem得到对应的header信息拷贝过来。

前端自动化测试(三)- angular和protracor

protractor用于前端UI自动化测试,特别为angular程序定制

特点

  • 端对端(e2e)测试
  • 采用jasmine作为测试框架
  • 基于WebDriverJS,(selenium-webdriver)
  • 针对angular应用增加定位器,更加方便实用
  • 实现自动等待,告别sleep wait,变异步为同步
  • 支持测试代码的调试
  • 支持多浏览器的并行UI测试

使用方法

准备工作

  • 1 安装protractor: npm install -g protractor
  • 2 安装selenium-standlone: webdriver-manager update
  • 3 启动selenium服务器: webdriver-manager start

spec书写

spec.js是用于书写测试用例的文件, protractor默认使用jasmine作为测试框架,举最简单的例子来说,一个spec文件可以这样写,使用describe作为
测试程序”块”, it定义一个用例,expect作为断言,其中browser这个全局的变量,用于操作浏览器.

describe('Protractor Demo App', function() {
    it('should have a title', function() {
        browser.get('http://juliemr.github.io/protractor-demo/');
        expect(browser.getTitle()).toEqual('Super Calculator');
    });
});

运行

  • 配置conf.json
    在测试之前,我们需要建立一个conf.json的文件,在这个文件中,可以配置测试的相关内容,例如:
    • multiCapabilities:使用哪些浏览器测试
    • chromeOptions:chrome浏览器的运行参数(使用哪些插件等)
    • framework:使用哪种测试框架: cucumber macha 还是jasmine
    • specs:测试哪些文件
      详细的配置信息请参考
exports.config = {
   // directConnect: true,

    // Capabilities to be passed to the webdriver instance.
    //capabilities: {
    //    'browserName': 'chrome'
    //},
    multiCapabilities: [ {
        'browserName': 'chrome',
        //'chromeOptions': {
        //    'args': ['--load-extension=/opt/local/share/nginx/html/radar/tanxtag'],
        //}
    }],
    // Framework to use. Jasmine is recommended.
    framework: 'jasmine',

    // Spec patterns are relative to the current working directly when
    // protractor is called.
    specs: ['basic/demo_spec.js','basic/angular_spec.js'],

    // Options to be passed to Jasmine.
    jasmineNodeOpts: {
        defaultTimeoutInterval: 30000
    }
};
  • 运行测试程序
protractor conf.json

grunt运行测试

有时我们需要使用grunt来配置测试任务,下面就是使用grunt-concurrent 模块实现并行运行多浏览器(也可通过conf.json中配置multiCapabilities解决)测试程序的代码:

module.exports = grunt => {
    //This module will read the dependencies/devDependencies/peerDependencies/optionalDependencies in your package.json
    // and load grunt tasks that match the provided patterns.
    require('load-grunt-tasks')(grunt);
    grunt.initConfig({
        concurrent: {
            protractor_test: ['protractor-chrome', 'protractor-firefox', 'protractor-safari']
        },
        protractor: {
            options: {
                keepAlive: true,
                singleRun: false,
                configFile: "conf.js"
            },
            run_chrome: {
                options: {
                    args: {
                        browser: "chrome"
                    }
                }
            },
            run_firefox: {
                options: {
                    args: {
                        browser: "firefox"
                    }
                }
            },
            run_safari: {
                options: {
                    args: {
                        browser: "safari"
                    }
                }
            }
        }
    });

    grunt.registerTask('protractor-chrome', ['protractor:run_chrome']);
    grunt.registerTask('protractor-firefox', ['protractor:run_firefox']);
    grunt.registerTask('protractor-safari', ['protractor:run_safari']);
    grunt.registerTask('protractor-e2e', ['concurrent:protractor_test']);
};

调试测试程序

除了非常方便的运行机制,protractor还提供便捷的调试方式, 使用selenium-webdriver操纵浏览器的时候,调试是非常困难的,在这里protractor就提供调试方式
在代码中加入 browser.pause(); 并且在终端输入 “repl” 就可以使用WebDriver commands来调试程序了:

wd-debug> repl
> element
function (locator) {
    return new ElementArrayFinder(ptor).all(locator).toElementFinder_();
  }
> 

测试非angular的应用

protractor内置方法测试angular的程序,例如它会自动检测angular页面加载完毕才会执行测试程序,当测试非angular程序的时候需要:
– 1 使用 browser.driver 代替 driver
– 2 添加 browser.driver.ignoreSynchronization = true
参考

protractor的详细使用

在protractor中,有几大类用于测试代码,详情请见protractorAPI
– browser: 浏览器的操作
– element & by: 定位获取页面元素
– ExpectedConditions:用于页面操作的逻辑函数,一般同wait连用
– webdriver: selenium 原生的语法函数
– promise:selenium内置的promise方法

浏览器的操作-browser

常用操作代码如下:

  • browser.get:
  • browser.findElement
  • browser.switchTo().frame()
  • browser.executeScript:
  • browser.executeAsyncScript
  • browser.wait:
  • browser.sleep:

选择器- by & element

支持多源选择器
– by.css()
– by.id()
– by.xpath()
– by.name()
– by.tagName()
– by.model():angular专用
– by.binding():angular专用
– by.repeater():angular专用

通过element获取:element(by.id(‘frameId’))或者element.all(by.css(‘some-css’));
在非angular应用中使用browser.driver.findElement(by.id(‘frameId’))

ExpectedConditions

预定义了wait的条件,常用的有
– elementToBeClickable: 按钮可以点击
– presenceOf: 元素出现在dom中
– titleContains: title含有某个字符串
– visibilityOf: 某个元素显示

    var EC = protractor.ExpectedConditions;
    var button = $('#xyz');
    var isClickable = EC.elementToBeClickable(button);

    browser.get(URL);
    browser.wait(isClickable, 5000); //wait for an element to become clickable
    button.click();

综合实例


it ('test login error', function () { _driver.get('http://subway.simba.taobao.com/#!/login'); _driver.wait(protractor.until.elementLocated(by.css('.login-ifr')),1000).then(function (elem) { _driver.switchTo().frame(elem); _driver.findElement(by.name('TPL_username')).sendKeys('zhangmeng1986712'); _driver.findElement(by.name('TPL_password')).sendKeys('xxxxx'); _driver.findElement(by.id('J_SubmitStatic')).click(); _driver.sleep(1000); browser.driver.findElement(by.css('.error')).then(function (elem) { return elem.getInnerHtml().then(function(text) { expect(text).toMatch('密码和账户名不匹配'); }); }); }); });

page object pattern

page object的模式大家一定不陌生,通过合理的配置可以使测试代码更容易维护,举例来说可以这样:

//书写一个input操作类
var AngularHomepage = function() {
  var nameInput = element(by.model('yourName'));
  var greeting = element(by.binding('yourName'));

  this.get = function() {
    browser.get('http://www.angularjs.org');
  };

  this.setName = function(name) {
    nameInput.sendKeys(name);
  };

  this.getGreeting = function() {
    return greeting.getText();
  };
};
//测试代码
describe('angularjs homepage', function() {
  it('should greet the named user', function() {
    var angularHomepage = new AngularHomepage();
    angularHomepage.get();
    angularHomepage.setName('Julie');
    expect(angularHomepage.getGreeting()).toEqual('Hello Julie!');
  });
});

mobile端的测试

详情参考
这个例子是使用Appium作为server端进行测试的,由于selenium-webdriver不能直接联Appium, 所以需要使用wd-bridge进行折衷.

e2e测试程序设计准则

参考

参考代码

本文的参考代码见 Github

前端自动化测试之单元测试(二)—— react组件的测试工具jest

前面介绍了Polymer的测试框架web-components-tester, 今天来看看React团队出品的Jest.在此,特别感谢婆婆帮忙带宝宝才让我有时间继续书写文章.

Jest的功能

  • 内置jasmine
  • 内置mock函数 可以mock模块
  • 基于jsdom
  • 同步化书写异步代码
  • 真心赞一下简洁明了的API定义和用法 以及清晰的文档,确实让书写单元测试不再痛苦
  • 适用于commonJS模块的单元测试
  • 运行速度较慢

Jest用法

  • 安装: npm install jest-cli(需要node版本大于4)
  • 配置 package.json 如下
  • 运行: npm test
  • 调试(使用node-debug再浏览器中调试):node-debug –nodejs –harmony ./node_modules/jest-cli/bin/jest.js –runInBand tests/getUser-test.js
  • 运行单个文件 ./node_modules/jest-cli/bin/jest.js tests/getUser-test.js
{
  "name": "jest-test-examples",
  "version": "0.0.1",
  "dependencies": {},
  "scripts": {
    "test": "jest"
  }
}

下面就具体介绍一下使用jest进行测试的方法,采用的例子为jest的官方实例

一个简单的测试

jest.dontMock('../src/sum');
describe('sum', function() {
    it('adds 1 + 2 to equal 3', function() {
        var sum = require('../src/sum');
        expect(sum(1, 2)).toBe(3);
    });
});

describe和it还有expect都使用了jasmine的语法, jest会自动mock所有的依赖模块,对模块中所有的输出方法予以遍历并进行mock,对于要测试的模块使用jest.dontMock标识,jest就不会去mock.

异步的单元测试实例

异步是javascript的灵魂, 所以异步的测试也是极其重要的,下面看看jest关于异步程序的测试,假如有这样个ajax程序,获取数据并进行解析,对其进行测试

  • 1 ajax的数据获取是否正确
  • 2 parseUserJson是否正确
    在第二个测试用例中使用了genMockFunction,用来对回调函数进行mock,在jest中有两种方式进行函数的mock
    • 1 使用xFunc = require(‘xx’);
    • 2 使用xFunc = jest.genMockFunction();
      使用后,会在xFunc.calls中存储有关函数的调用信息,例如
    • //mock.calls.length 记录了函数被调用了几次
    • //mock.calls[0][0] 被调用函数的第一个参数
    • //mock.calls[0][1] 第二个参数
      代码如下: $.ajax是一个被mock的函数,callback也被mock,getUser(callback)调用后,可以通过检测传递的参数判断是否正确.
var $ = require('jquery');
function parseUserJson(userJson) {
    return {
        loggedIn: true,
        fullName: userJson.firstName + ' ' + userJson.lastName
    };
}
function fetchCurrentUser(callback) {
    return $.ajax({
        type: 'GET',
        url: 'http://example.com/currentUser',
        success: function(userJson) {
            callback(parseUserJson(userJson));
        }
    });
}
module.exports = fetchCurrentUser;
jest.dontMock('../src/getUser');
describe('getUser', function() {
    //test right params
    it('calls into $.ajax with the correct params', function() {
        var $ = require('jquery');
        var getUser = require('../src/getUser');
        function dummyCallback() {}
        getUser(dummyCallback);
        // Now make sure that $.ajax was properly called during the previous
        // 2 lines
        expect($.ajax).toBeCalledWith({
            type: 'GET',
            url: 'http://example.com/currentUser',
            success: jasmine.any(Function)
        });
    });

    //test callback function
    it('calls the callback when $.ajax requests are finished', function() {
        var $ = require('jquery');
        var getUser = require('../src/getUser');

        //create mock function
        var callback = jest.genMockFunction();
        getUser(callback);

        //xfunc.mock have interactions information
        //mock.calls.length call times
        //mock.calls[0][0] first param
        //mock.calls[0][1] second param
        //https://facebook.github.io/jest/docs/mock-functions.html

        //emulate the params pass to success
        $.ajax.mock.calls[0][0].success({
            firstName: 'Bobby',
            lastName: '");DROP TABLE Users;--'
        });

        expect(callback.mock.calls[0][0]).toEqual({
            loggedIn: true,
            fullName: 'Bobby ");DROP TABLE Users;--'
        });

    });
});

React组件的单元测试实例

假如我们有这样一个checkbox react组件, 如下, react的es6写法请参考我的blog-ES6的核心语法与应用
原理非常简单,点击checkbox切换label的状态.我们的测试代码如下, 使用了react-addons-test-utils这个模块, 模块的renderIntoDocument用于将react组件渲染到document中,
并且支持产生模拟事件:TestUtils.Simulate.change.

import React from 'react';
class Checkbox extends React.Component {
    constructor(props) {
        super(props);
        this.state = {isChecked: false};
        this.changeState = this.changeState.bind(this);
    }
    changeState () {
        this.setState({isChecked: !this.state.isChecked})
    }
    render() {
        return (<label>
                <input type="checkbox" checked={this.state.isChecked} onChange={this.changeState} />
                {this.state.isChecked ? this.props.labelOn : this.props.labelOff}
               </label>)
    }

}
export default Checkbox;
import React from 'react';
import TestUtils from 'react-addons-test-utils';
import ReactDom from  'react-dom';
jest.dontMock('../src/checkbox');
const Checkbox = require('../src/checkbox');

describe('test react checkbox component', () => {
    it('change the label after click', () => {
        //1 render component
        //2 get node label get default value off
        //3 simulate click
        //4 expect value equal on

        //TestUtils.renderIntoDocument method refers to https://facebook.github.io/react/docs/test-utils.html#renderintodocument
        //TestUtils.findRenderedDOMComponentWithTag https://facebook.github.io/react/docs/test-utils.html#findrendereddomcomponentwithtag
        //ReactDom API:findDOMNode render unmountComponentAtNode server-side:renderToString renderToStaticMarkup

        var checkbox = TestUtils.renderIntoDocument(<Checkbox labelOn="On" labelOff="Off" />);
        var checkboxNode = ReactDom.findDOMNode(checkbox);
        //https://facebook.github.io/jest/docs/api.html#expect-value
        expect(checkboxNode.textContent).toEqual('Off');
        TestUtils.Simulate.change(TestUtils.findRenderedDOMComponentWithTag(checkbox, 'input'));
        expect(checkboxNode.textContent).toEqual('On');
    });
});

运行的时候我们需要通过babel预处理一下,通过如下的方式配置package.json即可运行:

  "scripts": {
    "test": "jest"
  },
  "jest": {
    "scriptPreprocessor": "<rootDir>/node_modules/babel-jest",
    "unmockedModulePathPatterns": [
      "<rootDir>/node_modules/react",
      "<rootDir>/node_modules/react-dom",
      "<rootDir>/node_modules/react-addons-test-utils"
    ],
    "modulePathIgnorePatterns": [
      "<rootDir>/node_modules/"
    ]
  }

手动mock

经常我们需要模拟某个模块中的方法(此方法实现非常复杂依赖第三方的模块)用来测试另一个模块的输入输出是否正确,jest就提供非常方便的mock机制,例如,我们在A模块中依赖jquery的fx方法
而fx方法又依赖于其他方法, 因为我们只关心fx的输出,所以我们就可以直接用来模拟,方法如下:

  • 建立mocks文件夹
  • 新建jquery模块:jquery.js
  • 使用genMockFromModule和mockImplementation API 如下
// mock the module of real jquery

var jqueryMocks = jest.genMockFromModule('jquery');
var mock_fx = function () {
    return 'mockValue';
};
//using mock_fx to mock the function of real fx
jqueryMocks.fx.mockImplementation(mock_fx);
module.exports = jqueryMocks;

这样就可以在测试代码中直接引用已经模拟好的fx函数进行测试了,直接对模块的输入控制,减少了依赖,实现测试的”解耦”.

describe('jest mocks', function () {
   it('jquery mock getEnv value', function(){
       var value = require('jquery').fx();
       expect(value).toEqual('mockValue')
   });
});

代码参考

源码

ES6的核心语法与应用

一直以来都对ES6嗤之以鼻,本来灵活简单的Javascrit,非得为了提升B格,增加学习的成本,搞那么多鸡肋的语法。但是无奈俺们这些“老年jser”都被历史的车轮碾压了,现在如果不掌握ES6,估计很多代码都看不懂了。没有闲暇的午后时间来系统的学习ES6(其实还是有点抵触心理),但是为了跟上“年轻人”的步伐,随着用随着看随着学吧。力求以最简单的语言讲述。

模块

定义模块

语法为:

  • export function x () {}
  • export class
  • export default {}
// kittydar.js - 找到一幅图像中所有猫的位置
    export function detectCats(canvas, options) {
      var kittydar = new Kittydar(options);
      return kittydar.detectCats(canvas);
    }
    export class Kittydar {
      ... 处理图片的几种方法 ...
    }
    // 这个helper函数没有被export。
    function resizeCanvas() {
      ...
    }
    //默认的
    export default {
        xx: '111'
    }

引用模块

 import {detectCats} from "kittydar.js"; //引入某个方法
 import {detectCats, Kittydar} from "kittydar.js"; //引入并重命名
 import * as module from './module';//引入全部全部
 import helloWorld from './hello-world'; //引入默认
 function go() {
        var canvas = document.getElementById("catpix");
        var cats = detectCats(canvas);
        drawRectangles(canvas, cats);
    }

class用法

  • 基本语法 class A {}
  • 构造器 constructor {}
  • 继承 class A extends AParent {}
  • super()
  • 注意:类声明与函数声明不同,它不会被提升,所以先new 后class定义 会抛出异常
  • 静态变量:static compare(a, b) {}
//ES5
//使用Object.defineProperty实现可读属性make year
function Vehicle(make, year) {
  Object.defineProperty(this, 'make', {
    get: function() { return make; }
  });

  Object.defineProperty(this, 'year', {
    get: function() { return year; }
  });
}

Vehicle.prototype.toString = function() {
  return this.make + ' ' + this.year;
}

var vehicle = new Vehicle('Toyota Corolla', 2009);
console.log(vehicle.make); // Toyota Corolla
vehicle.make = 'Ford Mustang'; //静态属性
console.log(vehicle.toString()) // Toyota Corolla 2009
//ES6
class Vehicle {
  constructor(make, year) {
    this._make = make;
    this._year = year;
  }

  get make() {
    return this._make;
  }

  get year() {
    return this._year;
  }

  toString() {
    return 'xxx';
  }
}

var vehicle = new Vehicle('Toyota Corolla', 2009);

console.log(vehicle.make); // Toyota Corolla
vehicle.make = 'Ford Mustang';
console.log(vehicle.toString()) // Toyota Corolla 2009
//ES5的继承
function Motorcycle(make, year) {
  Vehicle.apply(this, [make, year]);
}

Motorcycle.prototype = Object.create(Vehicle.prototype, {
  toString: function() {
    return 'xxx';
  }
});

Motorcycle.prototype.constructor = Motorcycle;

//ES6
class Motorcycle extends Vehicle {
  constructor(make, year) {
    super(make, year);
  }

  toString() {
    return 'xxxx';
  }
}

箭头函数

  • 箭头函数的产生,主要由两个目的:更简洁的语法和与父作用域共享关键字this。
  • function和{}都消失了,所有的回调函数都只出现在了一行里。
  • 当只有一个参数时,()也消失了(rest参数是一个例外,如(…args) => …)。
  • 当{}消失后,return关键字也跟着消失了。单行的箭头函数会提供一个隐式的return(这样的函数在其他编程语言中常被成为lamda函数)。
  • 箭头函数没有它自己的this值,箭头函数内的this值继承自外围作用域。
  • 箭头函数与普通函数还有一个区别就是,它没有自己的arguments变量,但可通过rest参数获得。
function () { return 1; }
() => { return 1; }
() => 1

function (a) { return a * 2; }
(a) => { return a * 2; }
(a) => a * 2
a => a * 2

function (a, b) { return a * b; }
(a, b) => { return a * b; }
(a, b) => a * b

function () { return arguments[0]; }
(...args) => args[0]

() => {} // undefined
() => ({}) // {}
//在之前的js中setInterval会把this指向window,
//使用箭头函数this使用外层的作用域所以不用保存this指针
$('.current-time').each(function () {
  setInterval(() => $(this).text(Date.now()), 1000);
});
//箭头函数的arguments,通过rest函数可以获得
function log(msg) {
  const print = (...args) => console.log(args[0]);
  print(`LOG: ${msg}`);
} 
log('hello'); // LOG: hello

作用域

javascript本身是没有块级作用域的,ES6新增的let语法替代var实现了块级作用域。

//before
function func(arr) {
    for (var i = 0; i < arr.length; i++) {
        // i ...
    }
    // 这里也可以访问到i
}
//ES6
function func(arr) {
    for (let i = 0; i < arr.length; i++) {
        // i ...
    }
    // 这里访问不到i
}

React on ES6

详情参考 这篇文章

定义组件

// The ES5 way
var Photo = React.createClass({
  handleDoubleTap: function(e) { … },
  render: function() { … },
});
// The ES6+ way
class Photo extends React.Component {
  handleDoubleTap(e) { … }
  render() { … }
}

componentWillMount关键字

// The ES5 way
var EmbedModal = React.createClass({
  componentWillMount: function() { … },
});
// The ES6+ way
class EmbedModal extends React.Component {
  constructor(props) {
    super(props);
    //实现componentWillMount内容的地方像dom操作
  }
}

state和props初始化

// The ES5 way
var Video = React.createClass({
  getDefaultProps: function() {
    return {
      autoPlay: false,
      maxLoops: 10,
    };
  },
  getInitialState: function() {
    return {
      loopsRemaining: this.props.maxLoops,
    };
  },
  propTypes: {
    autoPlay: React.PropTypes.bool.isRequired,
    maxLoops: React.PropTypes.number.isRequired,
    posterFrameSrc: React.PropTypes.string.isRequired,
    videoSrc: React.PropTypes.string.isRequired,
  },
});

// The ES6+ way
// static 实现只读的props
// 全局state
class Video extends React.Component {
  static defaultProps = {
    autoPlay: false,
    maxLoops: 10,
  }
  static propTypes = {
    autoPlay: React.PropTypes.bool.isRequired,
    maxLoops: React.PropTypes.number.isRequired,
    posterFrameSrc: React.PropTypes.string.isRequired,
    videoSrc: React.PropTypes.string.isRequired,
  }
  state = {
    loopsRemaining: this.props.maxLoops,
  }
}

react中的事件


class PostInfo extends React.Component { constructor(props) { super(props); // Manually bind this method to the component instance... this.handleOptionsButtonClick = this.handleOptionsButtonClick.bind(this); } handleOptionsButtonClick(e) { // this应指向实例 this.setState({showOptionsModal: true}); } } //箭头函数this指向外层的组件 class PostInfo extends React.Component { handleOptionsButtonClick = (e) => { this.setState({showOptionsModal: true}); } }

实例

使用ES6改写的React组件程序

import React from 'react'
class Checkbox extends React.Component {
    constructor(props) {
        super(props);
        this.state = {isChecked: false};
        this.changeState = this.changeState.bind(this);
    }
    changeState () {
        this.setState({isChecked: !this.state.isChecked})
    }
    render() {
        return (<label>
            <input type="checkbox" checked={this.state.isChecked} onChange={this.changeState} />
            {this.state.isChecked ? this.props.labelOn : this.props.labelOff}
        </label>)
    }

}
export default Checkbox;

hadoop mapreduce set number of mappers

As a hadoop platform developer, my job is to maintain the hadoop platform , fix the potential bugs and develop the new feature to meet my company’s need.I don’t get involve in the specfic mr job development.But today,one of my college asked me if i know how to set the map number in mapreduce job, it make to search for a while.
We all know, we can set the reduce number by call setNumReduceTasks.However,when you use setNumMapTasks to set numbers of mappers,it does not work,because this configration is just a hint,the inputformat just ignore it and create the splits for the job.
So i just check the code in FileInputFormat.class, and found how to set number of mappers in mapreduce job.
1.For files which file size is larger then block size

ie. block size is 128MB,and file size is several GB, we can set the mapreduce.input.fileinputformat.split.minsize paramter in configuration class.
Configuraiton conf = getConf();
conf.set("mapreduce.input.fileinputformat.split.minsize",1374389534720);
this make split size to be 1GB.

2.For small files which file size is smaller then block size

ie the file size is several MB,we have to use the CombineTextInputFormat replace the FileInputFormat, and use mapreduce.input.fileinputformat.split.minsize with the inputformat together to set number of mappers.
job.setInputFormatClass(CombineSmallfileInputFormat.class);
Configuraiton conf = getConf();
conf.set("mapreduce.input.fileinputformat.split.minsize",1374389534720);
Then you can control the number of mappers in you mr programme.

Socket Read Timeout 引发的问题

线上一个流式向hadoop集群写入数据的系统需要保证写入exactly once,并且不能丢失数据。为了解决这一问题,我们采用了thrift+write ahead log来构建这一系统。在使用过程中一直效果很好,最近偶然间发现这个服务发送了重复的日志。
经过检查发现,原来是是thrift tsocket read timeout导致的问题。我们的系统逻辑是client端向该系统发送日志,如果抛出异常,会将日志落到本地,等待一段时间后从本地读出日志继续发送。检查日志后发现在client端抛出

java.net.SocketTimeoutException: Read timed out

这就解释了写入重复的问题,首先client通过thrift TSokcet向server发送日志,如果超时了,client端timeout,抛出异常,将该条日志写入本地,过一段时间重现发送。而server端由于处理繁忙,没有及时响应client,虽然client中断连接,但是Server接收到该日志,并进行了处理。过一段时间后client端重新发送就产生了重复日志。
通过脚本检查这一问题的严重性,大概一条最多只有一条,目前将client端的超时时间增大解决这一问题。

YARN-4493 ResourceManager moveQueue bug

When moving a running application to a different queue in resourceManger , the current implement don’t check if the app can run in the new queue before remove it from current queue. So if the destination queue is full, the app will throw exception, and don’t belong to any queue.
Here is the code

   private void executeMove(SchedulerApplication app, FSSchedulerApp attempt,
       FSLeafQueue oldQueue, FSLeafQueue newQueue) {
     boolean wasRunnable = oldQueue.removeApp(attempt);
     // if app was not runnable before, it may be runnable now
     boolean nowRunnable = maxRunningEnforcer.canAppBeRunnable(newQueue,
         attempt.getUser());
     if (wasRunnable && !nowRunnable) {
       throw new IllegalStateException("Should have already verified that app "
           + attempt.getApplicationId() + " would be runnable in new queue");

After that, the queue become orphane, can not schedule any resources. If you kill the app, the removeApp method in FSLeafQueue will throw IllealStateException of “Given app to remove app does not exist in queue …” exception.
So i think we should check if the destination queue can run the app before remove it from the current queue.
The patch is below:

   private void executeMove(SchedulerApplication app, FSSchedulerApp attempt,
       FSLeafQueue oldQueue, FSLeafQueue newQueue) {
-    boolean wasRunnable = oldQueue.removeApp(attempt);
     // if app was not runnable before, it may be runnable now
     boolean nowRunnable = maxRunningEnforcer.canAppBeRunnable(newQueue,
         attempt.getUser());
+    boolean wasRunnable = false;
+    if (nowRunnable) {
+        wasRunnable = oldQueue.removeApp(attempt);
+    }
     if (wasRunnable && !nowRunnable) {
       throw new IllegalStateException("Should have already verified that app "
           + attempt.getApplicationId() + " would be runnable in new queue");