前端自动化测试之单元测试(一)—— polymer组件的测试工具

单元测试的工具们

  • 单元测试框架
    • Qunit – jquery的单测工具
    • jasmine – 早期的测试框架
    • mocha – 常用框架 支持BDD和TDD
    • Cucumber – 语义化更好的测试工具
  • 断言
    • chai
  • 集成
    • Karma
    • jenkins
    • travis-ci
  • mock
    • sinon
    • supertest
  • 组件测试框架
    • polymer的单测工具:web-component-tester
    • react的单测工具:Jest

之前的博客对比较基础的测试工具都有所介绍,参考前端自动化测试基础篇,在这篇blog中,我们首先了解一下polymer组件的单元测试工具。
mocha
chai
sinon

why web-component-tester

web-component-tester是polymer组件用于单元测试的框架,主要是用作对于html文件的测试
– 它内部集成了mocha、sinon、sinon-chai、chai,方便使用
– 使用lodash作为工具函数
– async用作异步函数的测试
– test-fixture作为

API方法

基本API

  • WCT.loadSuites可以将
  • suite类似describe将测试进行归类
  • test类似it进行具体的测试
  • assert用于进行断言,断言同chai的assert的断言规则
suite('AwesomeLib', function() {
  test('is awesome', function() {
    assert.isTrue(AwesomeLib.awesome);//TDD模式
  });
});

特殊的方法

  • text-fixture 用来在测试过程中操作template中的dom元素,用法是在
<test-fixture id="simple">
  <template>
    <div></div>
  </template>
</test-fixture>
<script>
  suite('classList', function() {
    var div;
    setup(function() {
      div = fixture('simple');
    })
    test('foo', function() {
      div.classList.add('foo');
      assertSomethingOrOther(div);
    });
  });
</script>

关于template binding异步的测试

异步测试在javascript的世界中可谓最常见,对于polymer组件来说,数据驱动模板刷新很重要,
模板数据的刷新会调用Polymer.dom.flush, 它是个异步的过程,对此 web-components-tester 专门提供
flush函数处理此类异步。

suite('with two selected items', function() {
  // Clean up after ourselves.
  teardown(function(done) {
    s.clearSelection();
    s.multi = false;
    // Wait for observers to resolve before moving on to more tests.
    flush(done);
  });

  test('multi selects by index', function(done) {
    s.multi = true;
    //数据变化
    s.selected = [0, 2];
    flush(function() {
      //模板刷新
      assert.equal(s.selectedIndex, [0, 2]);
      assert(s.children[0].classList.contains('core-selected'));
      assert(!s.children[1].classList.contains('core-selected'));
      assert(s.children[2].classList.contains('core-selected'));
      done();
    });
  });

});

google map component测试实例

  • 代码参考,f2e test
  • 1、安装 web-components-tester: npm install -g web-component-tester
  • 2 建立test文件夹(默认地址)
    • 2.1 index.html
    • 2.2 google-map-marker.html
    • 2.3 marker.js

suite('markers default', function () { var map; setup(function () { map = document.querySelector('#map'); }); test('markers are initialized', function () { var markerEl = Polymer.dom(map).querySelector('google-map-marker'); assert.isUndefined(markerEl.marker); assert.isUndefined(markerEl.map); assert.isNull(markerEl.info); assert.equal(markerEl.latitude, 37.779); assert.equal(markerEl.longitude, -122.3892); }); test('markers are added to map', function () { map.addEventListener('google-map-ready', function () { var mapMarkerEl = Polymer.dom(map).querySelector('google-map-marker'); var firstMarker = map.markers[0]; expect(firstMarker).to.deep.equal(mapMarkerEl); assert.equal(map.markers.length, 3); }); }); test('markers position can be updated', function (done) { map.addEventListener('google-map-ready', function (e) { var markerEl = Polymer.dom(map).querySelector('google-map-marker'); markerEl.latitude = 37.79493; markerEl.longitude = -122.41942; markerEl.zIndex = 1; assert.equal(markerEl.map, map.map, "marker's map is not the google-map's"); //重新渲染 异步过程 Polymer.dom.flush(); async.nextTick(function () { var marker = markerEl.marker; assert.equal(marker.getPosition().lat(), markerEl.latitude); assert.equal(marker.getPosition().lng(), markerEl.longitude); assert.equal(marker.getZIndex(), markerEl.zIndex); done(); }); }); }); });
  • 3 运行测试脚本 wct 即可。

wct运行机制

代码:runner
– 建立webserver,模板参见index.html
– 读取wct.conf.json配置
– 内置selenium server 通过wd.js建立连接打开浏览器进行测试
– 并将结果通过socketIO返回显示在命令行
– 测试框架核心:browser.js, 源码内置chai mocha sinon socket等以及polymer测试的辅助函数

辅助工具

在测试中少不了点击事件的模拟,wct这个工具不具有这个功能,但是可以使用polymer的工具组件iron-test-helpers
它内置了MockInteraction可以实现各个事件的模拟,只需import iron-test-helpers.html 即可。使用方法如下:

test('can be triggered with space', function(done) {
  button.addEventListener('keydown', function() {
    done();
  });
  MockInteractions.pressSpace(button);
});

test('can be clicked', function(done) {
  button.addEventListener('click', function() {
    done();
  });
  MockInteractions.tap(button);
});

Change LogLevel For MRAppMaster

Sometimes we want to show the debug log in MRAppMaster, there are two methods to do it. The first one is to change the mapred-site.xml in your gateway where you submit the job, add this conf.

<property>
<name>yarn.app.mapreduce.am.log.level</name>
<value>DEBUG</value>
</property>

The second one is to add config in the submit command like this

hadoop jar /usr/local/hadoop-2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar terasort -Dmapred.reduce.tasks=50 -Dmapreduce.map.speculative=false -Dmapreduce.reduce.speculative=false -Dyarn.app.mapreduce.am.log.level=DEBUG /test/1001 /test/1001_SORT14

前端自动化测试基础-sinon篇章

sinon用途

  • 在测试领域 Test double是很重要的一个概念。Test double主要用在自动化测试领域,会使用简单的对象或者流程模拟对应的行为减少测试的复杂性。
  • 用于 JavaScript 的测试监视(spy)、桩(stub)和仿制(mock)功能。不依赖其他类库,兼容任何单元测试框架。

sinon spy(最常用)

  • test spy 是这样的一类函数,它可以记录自己被调用的情况,包括传入的参数、返回结果、this 指向和抛出的错误(如果有的话)。test spy 可以是一个匿名函数,也可以是对一个已有函数进行的封装。
  • 用于测试callback函数
  • 用于spy已知行为的方法
  • 创建spy
    //创建一个匿名的函数用于记录调用的参数、返回值、以及异常
    var spy = sinon.spy();
    var spy = sinon.spy(myFunc);
    //对对象的方法增加spy 用于替换原有方法的行为,可以通过调用object.method.restore()实现恢复设置
    var spy = sinon.spy(object, "method");
  • sinon API用法
    • 判断某函数调用了某些参数:spy.withArgs(arg1[, arg2, …]); e.g. assert(spy.withArgs(42).calledOnce);
    • 某函数调用的次数:spy.callCount
    • spy.called
    • spy.calledTwice spy.calledThrice
    • spy.firstCall spy.secondCall spy.thirdCall
    • 判断是否在另一个spy之前(后)被调用 spy.calledBefore(anotherSpy);spy.calledAfter(anotherSpy);
    • 至少有一次被某个参数调用,参数可以部分匹配:spy.calledWith(arg1, arg2, …);
    • 至少有一次抛出异常:spy.threw();
  • sinon spy实例
        //以backbone的Event单测为例
        var eventer = _.extend({}, Backbone.Events),
            spy = sinon.spy();

        // Set up the spy.
        eventer.on("foo", spy);
        expect(spy.called).to.be.false;

        // Fire event.
        eventer.trigger("foo", 42);

        // Check number of calls.
        expect(spy.calledOnce).to.be.true;
        expect(spy.callCount).to.equal(1);

        // Check calling arguments.
        expect(spy.firstCall.args[0]).to.equal(42);
        expect(spy.calledWith(42)).to.be.true;

sinon stub

  • stub(桩)其实是最抽象最难理解的,Test stubs是一类预编码行为的函数(也是一种 spy)。除了改变stub对象的行为之外,它还支持所有的 spy API。同spy一样,stubs 可以是匿名函数,或者包装已有函数。当使用 stub 包装一个已有函数时,原函数将不会被调用。
  • stub用于:
    • 在测试中控制一个方法的行为,以强制代码沿特定路径执行。例如测试错误处理时,可以强制一个方法抛出错误。
    • 当你希望阻止一个方法被直接调用时(可能是因为这个方法触发了干扰行为,例如 XHR 请求之类的)。
  • 创建stub

    • 创建一个匿名的 stub 函数。var stub = sinon.stub();
    • 使用一个 stub 函数替代 object.method。原函数可以通过调用 object.method.restore() (或 stub.restore())方法来还原。如果 object.method 不是一个函数,则会抛出一个异常来帮助你避免类型错误。var stub = sinon.stub(object, “method”);
    • 使用 func 来替换 object.method,并且被包装在一个 spy 中。object.method.restore() 可以恢复原方法。var stub = sinon.stub(object, “method”, func);
    • stub 该对象的所有方法。var stub = sinon.stub(obj);
  • stub API,详情
    • stub.withArgs(arg1[, arg2, …]);
    • stub.returns(obj);
    • stub.throws(); 例: var callback = sinon.stub(); callback.withArgs(1).throws(“TypeError”);
    • stub.yieldsTo(property, [arg1, arg2, …])
  • sinon stub实例
//basic usage
    var obj = {
      multiply: function (a, b) { return a * b; },
      error: function (msg) { throw new Error(msg); }
    };

    it("stubs multiply", function () {
      // Stub with a hard-coded return value.
      sinon.stub(obj, "multiply").returns(5);
      expect(obj.multiply(1, 2)).to.equal(5);
      obj.multiply.restore();

      // Stub with a function.
      sinon.stub(obj, "multiply", function (a, b) {
        return a + b;
      });
      expect(obj.multiply(1, 2)).to.equal(3);
      obj.multiply.restore();
    });

    it("stubs error", sinon.test(function () {
      this.stub(obj, "error");
      expect(obj.error).to.not.throw();
    }));
  });

//use yieldsTo
    it("stubs with yieldsTo", function () {
            var obj = {
                    async: function (opts) {
                        opts.success("a", "b");
                    }
                },
                spyObj = {
                    failure: sinon.spy(),
                    success: sinon.spy()
                };

            sinon.stub(obj, "async").yieldsTo("success", 1, 2);

            // Call on object with callback spies.
            obj.async(spyObj);

            expect(spyObj.failure).to.have.not.have.been.called;
            expect(spyObj.success)
                .to.have.been.calledOnce.and
                .to.have.been.calledWith(1, 2);
        });

##sinon mock

  • 用于给出expectation然后验证某个object的method是否是正确的
  • 同spy的区别,mock出的object收到了数据或是调用并没有真正执行,一切针对mock的调用都是假的。所以mock可以用来测试具有side effect的函数,这里的side effect泛指和外部对象有数据交互或者是调用,比如调用外部对象的方法、向server发送数据、和UI对象有交互、写日志等等。
  • API
    • 创建mock:var mock = sinon.mock(obj);
    • 给出expectation mock.expects(“method”);
    • 校验是否正确: mock.verify();
    • 重置:mock.restore();
  • sinon mock 例子

describe("Sinon.JS mocks", function () {
  // Object literal with two methods.
  var obj = {
    multiply: function (a, b) { return a * b; },
    error: function (msg) { throw new Error(msg); }
  };

  it("mocks multiply", function () {
    // Create the mock.
    var mock = sinon.mock(obj);

    // The multiply method is expected to be called:
    mock.expects("multiply")
      .atLeast(2)    // 2+ times,
      .atMost(4)     // no more than 4 times, and
      .withArgs(2);  // 2 was first arg on *all* calls.

    // Make 3 calls to `multiply()`.
    obj.multiply(2, 1);
    obj.multiply(2, 2);
    obj.multiply(2, 3);

    // Verify **all** of the previous expectations.
    mock.verify();

    // Restore the object.
    mock.restore();
  });

});

sinon Fake XMLHttpRequest/ Fake Server

  • fake server用法
    {
        setUp: function () {
            this.server = sinon.fakeServer.create();//创建server
        },

        tearDown: function () {
            this.server.restore();
        },

        "test should fetch comments from server" : function () {
            this.server.respondWith("GET", "/some/article/comments.json",
                [200, { "Content-Type": "application/json" },
                 '[{ "id": 12, "comment": "Hey there" }]']);

            var callback = sinon.spy();
            myLib.getCommentsFor("/some/article", callback);
            this.server.respond();

            sinon.assert.calledWith(callback, [{ id: 12, comment: "Hey there" }]);
        }
    }

  • sinon可以用作实现request的模拟,现在更多的使用supertest
    用于HTTP的测试
describe('GET /user', function(){
  it('user.name should be an case-insensitive match for "tobi"', function(done){
    request(app)
      .get('/user')
      .set('Accept', 'application/json')
      .expect(function(res) {
        res.body.id = 'some fixed id';
        res.body.name = res.body.name.toUpperCase();
      })
      .expect(200, {
        id: 'some fixed id',
        name: 'TOBI'
      }, done);
  });
});

sinon 测试代码

DataXceiver本地读异常bug说明(HDFS-11802)

现象描述

用户在读取文件的时候报三台DN都无法取得该文件对应的block,经过fsck检查后没有发现该文件有丢块现象,到对应的dn上去查看日志,发现三台机器已经都处于不可读状态,报错为

2015-11-25 00:01:55,999 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 10.39.5.160:50010:DataXceiverServer:
java.io.IOException: Xceiver count 4097 exceeds the limit of concurrent xcievers: 4096
    at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:137)
    at java.lang.Thread.run(Thread.java:745)

很明显,这是超过了dataxceiver设置的最大的线程数4096,正常情况下是不可能超过的,所以说明dn有线程泄露的bug或者其它问题。
再检查日志,发现了以下的日志

Exception in thread "Thread-19" java.lang.IllegalStateException: failed to remove c53ce04928d1baa854f5dc1bfc8d565b
    at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
    at org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.removeShm(ShortCircuitRegistry.java:115)
    at org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry$RegisteredShm.handle(ShortCircuitRegistry.java:102)
    at org.apache.hadoop.net.unix.DomainSocketWatcher.sendCallback(DomainSocketWatcher.java:371)
    at org.apache.hadoop.net.unix.DomainSocketWatcher.access$1000(DomainSocketWatcher.java:52)
    at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:511)
    at java.lang.Thread.run(Thread.java:745)

看到这个日志以后就大概知道了原因,是由于DomainSocketWatcher线程异常退出,导致本地读线程没有回收机制,占满了所有的dataxceiver slot导致的。

问题原因

DomainSocketWatcher线程负责对本地读线程建立的socket进行一些处理和清理等。出问题的代码为

<code>
      try {
        while (true) {
              doSomecleanup.......
        }
      } catch (InterruptedException e) {
        LOG.info(toString() + " terminating on InterruptedException");
      } catch (IOException e) {
        LOG.error(toString() + " terminating on IOException", e);
      } finally {
        lock.lock();
        try {
          kick(); // allow the handler for notificationSockets[0] to read a byte
          Iterator<Entry> iter = entries.values().iterator();
          while(iter.hasNext()) {
            sendCallback("close", iter, fdSet);
          }
          entries.clear();
          fdSet.close();
        } finally {
          lock.unlock();
        }
      }
</code>

正常情况下代码不会走入到finally,而是一直在while中loop。而报出的异常则是remove一个共享内存对象的时候失败,而导致的运行时异常。
经过jira查询和我们集群中机器日志查看,发现原因为在于Client向DataNode申请本地读时候,DataNode建立共享内存对象以及File Descriptor出现异常,导致分配失败,日志为

2015-11-06 04:52:41,080 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: cliID: DFSClient_attempt_1435099124107_5925361_m_000028_0_1777694543_1, src: 127.0.0.1, dest: 127.0.0.1, op: REQUEST_SHORT_CIRCUIT_SHM, shmId: n/a, srvID: 01f352c6-4e63-4158-8ead-3e8146103b6f, success: false

而在DataXceiver的requestShortCircuitShm代码中,如果失败则close连接

      if ((!success) && (peer == null)) {
        // If we failed to pass the shared memory segment to the client,
        // close the UNIX domain socket now.  This will trigger the 
        // DomainSocketWatcher callback, cleaning up the segment.
        IOUtils.cleanup(null, sock);
      }

但是,所有close操作都是通过回调DomainSocketWatcher来做的,这样,当DomainSocketWatcher再次close的时候内存中的共享内存对象由于已经close被释放,而报runtimeerror,这样DomainSocketWatcher线程异常退出,本地读没有清理线程,慢慢占满了slot,最后导致了DataNode不可用。

解决方法

DataXceiver不负责close连接,而只是负责将与client连接shutdown,使client能够快速反应读异常,同时增加更多的catch,如果后续还有异常能够找到原因。

YARN MRAppMaster启动流程(一)

从这篇起,开始介绍MRAppmaster与NodeManager交互启动,以及申请资源等内容。

1.接YARN MRAppMaster与Scheduler流程说明,ApplicationMasterLauncher启动一个AMLaucher线程,用于启动MRAppmaster。首先是通过RPC连接,interface是ContainerManagementProtocol,负责ApplicationMaster向NodeManager通信,然后封装所有的运行环境,参数等到context中,随后就是通过RPC向NodeManager发送startContainer命令。

    StartContainersResponse response =
        containerMgrProxy.startContainers(allRequests);

至此,ResourceManager调度完成,下面的工作交给了NodeManager。

2.ContainerManagerImpl.startContainers
NodeManager是通过ContainerManagerImpl模块来处理ApplicationMasterLauncher的请求的,首先会对ugi,token进行验证,随后就启动container,获得启动的用户,执行的所有命令和conf,随后根据这些信息生成Application对象用来执行。

    Container container =
        new ContainerImpl(getConfig(), this.dispatcher, launchContext,
          credentials, metrics, containerTokenIdentifier);
    ApplicationId applicationID =
        containerId.getApplicationAttemptId().getApplicationId();
 
    Application application =
            new ApplicationImpl(dispatcher, user, applicationID, credentials, context);

随后通过NodeManager的dispatcher发送ApplicationEventType.INIT_APPLICATION事件。

3.ApplicationEventType.INIT_APPLICATION
ApplicationImpl处理ApplicationEventType.INIT_APPLICATION事件,通知logHandler处理

      // Inform the logAggregator
      app.dispatcher.getEventHandler().handle(
          new LogHandlerAppStartedEvent(app.appId, app.user,
              app.credentials, ContainerLogsRetentionPolicy.ALL_CONTAINERS,
              app.applicationACLs));

4.LogHandlerAppStartedEvent
我们集群开启了logAggregation,配置方法为在yarn-site.xml配置yarn.log-aggregation-enable为true。随后由LogAggregationService类进行处理,主要是初始化aggregator,代码为

  private void initApp(final ApplicationId appId, String user,
      Credentials credentials, ContainerLogsRetentionPolicy logRetentionPolicy,
      Map<ApplicationAccessType, String> appAcls) {
    ApplicationEvent eventResponse;
    try {
      verifyAndCreateRemoteLogDir(getConfig());
      initAppAggregator(appId, user, credentials, logRetentionPolicy, appAcls);
      eventResponse = new ApplicationEvent(appId,
          ApplicationEventType.APPLICATION_LOG_HANDLING_INITED);
    } catch (YarnRuntimeException e) {
      LOG.warn("Application failed to init aggregation", e);
      eventResponse = new ApplicationEvent(appId,
          ApplicationEventType.APPLICATION_LOG_HANDLING_FAILED);
    }
    this.dispatcher.getEventHandler().handle(eventResponse);
  }

随后向Dispatcher发送ApplicationEventType.APPLICATION_LOG_HANDLING_INITED事件。
5.ApplicationEventType.APPLICATION_LOG_HANDLING_INITED
由ApplicationImpl处理,代码为

  static class AppLogInitDoneTransition implements
      SingleArcTransition<ApplicationImpl, ApplicationEvent> {
    @Override
    public void transition(ApplicationImpl app, ApplicationEvent event) {
      app.dispatcher.getEventHandler().handle(
          new ApplicationLocalizationEvent(
              LocalizationEventType.INIT_APPLICATION_RESOURCES, app));
    }
  }

最后通过dispatcher发送ApplicationLocalizationEvent事件。
6. LocalizationEventType.INIT_APPLICATION_RESOURCES
在ContainerManagerImpl中注册了ApplicationLocalizationEvent的处理方法为ResourceLocalizationService,代码是

dispatcher.register(LocalizationEventType.class, rsrcLocalizationSrvc);

这时候开始了MRAppmaster本地化的过程,所谓本地化就是把执行需要的所有资源,环境从集群中下载到执行MRAppmaster的NodeManager中的相应目录中。关于这部分的内容,我推荐看hortonwokrs的这篇文章来仔细了解。
最后发送ApplicationInitedEvent事件到ApplicationImpl处理。到这一步,application初始化完毕,表示application在这台机器上运行,建立了相应的内存结构,下一步开始了初始化container。
7.ApplicationInitedEvent.APPLICATION_INITED
直接发送ContainerEventType.INIT_CONTAINER到distapcher。
8.ContainerEventType.INIT_CONTAINER
ContainerImpl来处理ContainerEventType事件,开始本地化过程。调用了RequestResourcesTransition函数来处理。这里会对所有的资源,根据PUBLIC,PRIVATE,APPLICATION分别处理放到 Map<LocalResourceVisibility, Collection> req 中,发送LocalizationEventType.INIT_CONTAINER_RESOURCES事件。
9.LocalizationEventType.INIT_CONTAINER_RESOURCES
对于7中不同类型的资源,转发给相应的tracker进行处理。处理事件为ResourceEventType.REQUEST
10.ResourceEventType.REQUEST
LocalizedResource来处理这个事件。开始下载过程。代码为

  private static class FetchResourceTransition extends ResourceTransition {
    @Override
    public void transition(LocalizedResource rsrc, ResourceEvent event) {
      ResourceRequestEvent req = (ResourceRequestEvent) event;
      LocalizerContext ctxt = req.getContext();
      ContainerId container = ctxt.getContainerId();
      rsrc.ref.add(container);
      rsrc.dispatcher.getEventHandler().handle(
          new LocalizerResourceRequestEvent(rsrc, req.getVisibility(), ctxt, 
              req.getLocalResourceRequest().getPattern()));
    }
  }

11.LocalizerResourceRequestEvent
由ResourceLocalizationService来处理,PUBLIC的资源由PublicLocalizer下载,PRIVATE和APPLICATION的资源由LocalizerRunner来处理。
PublicLocalizer内部有个queue为pending,做了同步,通过线程不断的拉文件系统的数据,通过FSDowload类来做。代码为

              pending.put(queue.submit(new FSDownload(lfs, null, conf,
                  publicDirDestPath, resource, request.getContext().getStatCache())),
                  request);

PRIVATE和APPLICATIOn则是通过LocalizerRunner线程不断拉取,最后有两种方式,一种是DefaultContainerExecutor一种是LinuxContainerExecutor(cgroup用来做资源隔离)来做,看一下默认的Default

//    ContainerLocalizer localizer =
//        new ContainerLocalizer(lfs, user, appId, locId, getPaths(localDirs),
//            RecordFactoryProvider.getRecordFactory(getConf()));
 
    createUserLocalDirs(localDirs, user);
    createUserCacheDirs(localDirs, user);
    createAppDirs(localDirs, user, appId);
    createAppLogDirs(appId, logDirs);
 
 
    //Path appStorageDir = getFirstApplicationDir(localDirs, user, appId);
    // randomly choose the local directory
    Path appStorageDir = getWorkingDir(localDirs, user, appId);
 
    String tokenFn = String.format(ContainerLocalizer.TOKEN_FILE_NAME_FMT, locId);
    Path tokenDst = new Path(appStorageDir, tokenFn);
    lfs.util().copy(nmPrivateContainerTokensPath, tokenDst);
    LOG.info("Copying from " + nmPrivateContainerTokensPath + " to " + tokenDst);
//    lfs.setWorkingDirectory(appStorageDir);
//    LOG.info("CWD set to " + appStorageDir + " = " + lfs.getWorkingDirectory());
    // TODO: DO it over RPC for maintaining similarity?
    FileContext localizerFc = FileContext.getFileContext(
        lfs.getDefaultFileSystem(), getConf());
    localizerFc.setUMask(lfs.getUMask());
    localizerFc.setWorkingDirectory(appStorageDir);
    LOG.info("Localizer CWD set to " + appStorageDir + " = " 
        + localizerFc.getWorkingDirectory());
    ContainerLocalizer localizer =
        new ContainerLocalizer(localizerFc, user, appId, locId, 
            getPaths(localDirs), RecordFactoryProvider.getRecordFactory(getConf()));
 
    localizer.runLocalization(nmAddr);

重要的是ContainerLocalizer,在runLocalization代码为

     exec = createDownloadThreadPool();
     CompletionService<Path> ecs = createCompletionService(exec);
     localizeFiles(nodeManager, ecs, ugi);

用线程池来下载集群中的资源到本地对应目录,并设置相应的权限。而localizeFiles则会周期性的向ResourceLocalizationService汇报状态,

         try {
            nodemanager.heartbeat(status);
          } catch (YarnException e) { }
          return;
        }
        cs.poll(1000, TimeUnit.MILLISECONDS);

在ResourceLocalizationService对应代码为

  @Override
  public LocalizerHeartbeatResponse heartbeat(LocalizerStatus status) {
    return localizerTracker.processHeartbeat(status);
  }

如果本地化全部完成,在processHeartbeat后会有状态机的改变

     case FETCH_SUCCESS:
            // notify resource
            try {
            getLocalResourcesTracker(req.getVisibility(), user, applicationId)
              .handle(
                new ResourceLocalizedEvent(req, ConverterUtils
                  .getPathFromYarnURL(stat.getLocalPath()), stat.getLocalSize()));

至此,资源本地化完成,container等待执行,此时event为ResourceLocalizedEvent。

响应式设计的自动化测试

响应式设计介绍

响应式设计(RWD)从2010年开始就逐渐进入人们的视线,虽然由于网速和网络的制约,目前国内都是采用针对pc和移动分别开发站点的策略,例如淘宝网的首页在pc端网页 ,在移动端的网页为网页,使用的是基于REM的布局设计。但是没人能否认响应式设计的重要性和简便性。响应式设计其实简单来说就是利用media query针对不同的设备和分辨率采用不同的css样式,用以达到网站在各个设备上的兼容性,再结合“移动优先”的策略,使得响应式设计更加的具有优势。如下图所示,就是一个简单的购物网站在不同设备上的响应式设计。

一个响应式设计的实现

下面就举一个简单的响应式网站的例子
代码见f2e-testing

  • 首页welcome页面
  • 登陆页面
  • notes列表页面
  • 新建note页面



公共头尾和菜单的响应式实现

这个属于很经典的响应式菜单和标题设计:
– 首页中如果是mobile或者ipad 则只显示标题的主要部分
– 菜单中如果是大屏,则一行显示菜单,mobile下用两行的菜单
– css完整实现:参见


<div id="header"> <div class="middle-wrapper"> <img id="header-logo" src="../src/assets/header-icon.png"> <h1> Sample Website <span class="not-on-mobile not-on-tablet">for Galen Framework</span></h1> </div> </div>
#menu {
    background: url("images/menu-background.png");
    margin: 0;
    color: white;
}
#menu ul {
    margin: 0;
    padding: 0;
    list-style: none;
}
#menu li {
    display: inline-block;
}
#menu ul:after {
    clear: both;
}
#menu li a {
    min-width: 100px;
    font-size: 1.2em;
    color: white;
    padding: 20px;
    display: inline-block;
}

@media (max-width: 500px) {
    .not-on-mobile {
        display: none;
    }

    #menu {
        width: 100%;
    }
    #menu li {
        width: 49%;
    }
    #menu li a {
        width: 100%;
    }
}
@media (max-width: 800px) {
    .not-on-tablet {
        display: none;
    }
}

welcome页面

welcome页面:使用的是bootstrap的jumbotron的布局,这款响应式布局
主要用在simple marketing or informational website。它具有一个通知的大型“布告栏”(jumbotron)和三栏式布局。
主要的实现代码:

    <div class="middle-wrapper">
        <div id="content">
            <div id="welcome-page" class="jumbotron">
                <h1>Welcome to our test page!</h1>

                <p>This app is used as a playground for <a href="http://galenframework.com/">Galen Framework</a></p>

                <p>
                    <button class="btn btn-lg btn-primary button-login" type="button" onclick="App.showLoginPage();">
                        Login
                    </button>
                </p>
                <p>To log in this website use the email <b>testuser@example.com</b> and password <b>test123</b>
                </p></div>
        </div>
    </div>

.jumbotron { padding: 30px; margin-bottom: 30px; color: inherit; background-color: #eee; } .jumbotron h1, .jumbotron .h1 { color: inherit; } .jumbotron p { margin-bottom: 15px; font-size: 21px; font-weight: 200; } .container .jumbotron { border-radius: 6px; } .jumbotron .container { max-width: 100%; } //大于768px @media screen and (min-width: 768px) { .jumbotron { padding-top: 48px; padding-bottom: 48px; } .container .jumbotron { padding-right: 60px; padding-left: 60px; } .jumbotron h1, .jumbotron .h1 { font-size: 63px; } } @media (max-width: 500px) { button { width: 100%; margin-top: 10px; } }

login登陆页面


@media (min-width: 501px) {
    .dialog-panel {
        width: 400px;
        border: 1px solid #ccc;
        padding: 20px;
        margin: auto;
        border-radius: 10px;
        box-shadow: 1px 3px 3px #ddd;
    }
}

响应式兼容尺寸

  • 合理的使用viewport

<meta name="viewport" content="width=device-width, initial-scale=1"> <!--或者使用如下的 --> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  • 使用hack兼容低版本浏览器的media query

<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries --> <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> <!--[if lt IE 9]> <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script> <script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script> <![endif]-->
  • 参考bootstrap 3 它优先使用“移动优先”原则,详情:参考
    • Extra small devices ~ Phones (< 768px) col-xs-
    • Small devices ~ Tablets (>= 768px) col-sm-
    • Medium devices ~ Desktops (>= 992px) col-md-
    • Large devices ~ Desktops (>= 1200px) col-lg-

/*========== Mobile First Method ==========*/ /* RWD is – Desktop -> Tablet -> Mobile */ /* Mobile First RWD is – Mobile -> Tablet -> Desktop */ /* Custom, iPhone Retina */ @media only screen and (min-width : 320px) { } /* Extra Small Devices, Phones */ @media only screen and (min-width : 480px) { } /* Small Devices, Tablets */ @media only screen and (min-width : 768px) { } /* Medium Devices, Desktops */ @media only screen and (min-width : 992px) { } /* Large Devices, Wide Screens */ @media only screen and (min-width : 1200px) { } /*========== Non-Mobile First Method ==========*/ /* Large Devices, Wide Screens */ @media only screen and (max-width : 1200px) { } /* Medium Devices, Desktops */ @media only screen and (max-width : 992px) { } /* Small Devices, Tablets */ @media only screen and (max-width : 768px) { } /* Extra Small Devices, Phones */ @media only screen and (max-width : 480px) { } /* Custom, iPhone Retina */ @media only screen and (max-width : 320px) { }

响应式设计的自动化测试框架 – galenframework

介绍

  • 用于响应式设计的开源UI自动化测试框架
  • 测试spec “语义化友好”,通过位置信息准确定位各个元素的位置
  • 测试用例API兼容java和javascript
  • pc端和无线端多尺寸兼容,支持selenium appium saucelab browserstack多服务器测试
  • 可自定义输出的测试 html report

安装

  • 下载二进制代码
  • 执行 ./install.sh
  • galen -v 显示如下命令行 表明安装成功
Galen Framework
Version: 2.1.2
JavaScript executor: Rhino 1.7 release 5 2015 01 29

测试环境建立

  • 执行 galen config:生成config文件用于配置初始化文件,具体参数配置 详情参见
  • 文件结构
    • tests文件夹:用于装载测试脚本
      • init.js: 用于配置测试的设备和尺寸
      • pages文件夹: ui自动化测试的Page Object页面
      • login.page.test.js(默认是以.test.js后缀作为测试文件,如果有特殊要求可以在config文件中配置)
    • specs文件夹: 用于装载响应式设计的规则spec文件
      • common.spec文件:
      • loginPage.spec文件等等
    • config文件:配置文件
    • reports目录:用于生成自动化测试的html结果

构建测试服务

  • appium作为mobile的测试服务器,android真机测试的服务搭建,参考
  • selenium作为pc端的测试服务器
#server端:8002端口启动三星galaxy SIII设备的测试服务器;8001端口启动IPAD模拟器;启动chromepc端的测试服务器
node . -a 127.0.0.1 -p 8002 -U 4df752b06833bfd3 --browser-name Chrome --no-reset
node . -a 127.0.0.1 -p 8001 --command-timeout 50000  --no-reset
selenium-standalone start

#客户端:测试 并且测试完成后浏览器打开测试结果

Galen的命令行运行,参考

  • galen check:运行spec
  • galen test: 运行测试用例
  • galen dump:生成可视化spec-
galen test mytest01.test
    --htmlreport "htmlreport-dir"
    --testngreport "report/testng.xml"
    --jsonreport "jsonreport-dir"
    --parallel-tests 4

galen test tests/ --htmlreport reports   

galen check homepage.gspec
    --url "http://example.com"
    --size "640x480"
    --javascript "some.js"
    --include "mobile,all"
    --exclude "toexclude"
    --htmlreport "htmlreport-dir"
    --testngreport "report/testng.xml"
    --jsonreport "jsonreport-dir"

galen dump "specs/homepage.gspec"
    --url "http://galenframework.com"
    --size "1024x768"
    --export "dumps/homepage-dump" 
    --max-width "200" 
    --max-height "200"    

测试流程

createGridDriver建立对服务器的链接,并启动driver


var driver = createGridDriver('http://127.0.0.1:8001/wd/hub',{ desiredCapabilities: { browserName: 'Safari', 'platformVersion': '9.1', 'platformName': 'iOS', 'app': 'safari', deviceName:"iPad Air", size: '600x800', safariInitialUrl: 'about:blank' } }); driver.get("http://test.xxxxx.com");

checkLayout连接spec文件和.test.js测试文件

  • 编写测试脚本
  • 编写spec文件

检查spec文件是否符合预期

//定义test
test("Simplest test", function () {
    // here goes a test code
});

//[] spec中 @on的tag名称
checkLayout(driver, "specs/welcomePage.spec", ['desktop']);

使用 Page Object Model

PageObject在selenium中是常见的设计模式,它可以快速的将测试用例和测试主体相互分开,通过复用,减少代码;同时可以把测试过程变化的参数在统一的地方配置,减少改动的成本。关于 Page Object我会再开文介绍,这里只为大家介绍在galenframework中我们可以如何快捷的定义我们的PageObject,以登陆页为参考:

$page(pageName, primaryFields, [ secondaryFields ])

this.LoginPage = $page("Login page", {
  email: "input.email", // css locator
  password: "xpath: //input[@class='password']", // xpath locator
  submitButton: "id: submit", // id locator

  load: function () {
    this.open("http://example.com/login");
    return this.waitForIt();
  },
  loginAs: function (userName, password) {
    this.email.typeText(userName);
    this.password.typeText(password);
    this.submitButton.click();
  }
});
// now you can use it like this
var loginPage = new LoginPage(driver).load();
loginPage.loginAs("testuser@example.com", "password");

页面的webdriver操作函数

参考 GalenPage.js

  • 针对于$page这个对象
    • open 打开页面
    • waitForIt 等到primaryFields的元素都
    • wait({}).untilAll({}); 等
    • getAllLocators:把
    • findChild 定位元素
    • findChildren 批量定位元素
  • 针对pageElement(primaryFields还有secondaryFields中的元素)
    • attribute 获取属性
    • cssValue 获得css属性值
    • typeText input输入内容
    • click 点击按钮
    • clear 清空input
    • getText 获得输入的内容
    • hover
    • getWebElement
    • findChild
    • findChildren
    • isDisplayed 元素是否展现

操作并输出到report中

  • logged(text, callback)
  • loggedFunction(textExpression, callback)

//${_1} ${_2} 代表arguments loggedFunction("Log-in as ${_1} with password ${_2}", function (email, password) { this.emailTextfield.typeText(email); this.passwordTextfield.typeText(password); this.submitButton.click(); });

spec文件编写

spec文件是用于描述元素css之间的各种关系,符合语义化的要求,详情 参考

  • 定义Objects:@objects
  • tags和sections:= Main section =
  • 变量 @set
  • import其他的spec文件规则:@import header.spec
  • forEach Loop:循环
  • near – checks that object is located near another object
  • below – checks that an element is located below other object
  • above – checks that an element is located above other object
  • left-of and right-of – checks that an element is located above other object
  • inside – checks that object is located inside another object
  • width – checks the width of object
  • height – checks the height of object
  • aligned – checks horizontal or vertical alignment of object with other objects on page
  • text – checks the text that is visible on page
    • text is – checks that text is exactly as expected
    • text contains -checks element contains expected text
    • text starts – element should start with expected text
    • text ends – element should end with expected text
    • text matches – verifies that text matches Java Regular Expression
  • centered – checks that object is centered inside another object
  • absent – checks that object is either missing on page or is not visible
  • contains – checks that object visually contains other objects inside it
  • on – checks that object is visually located on other object
  • component – runs a subset of specs from another file within the given object context
  • color-scheme – checks the color distribution in the given object area


@objects search-panel id search-bar search-panel-input xpath //div[@id='search-bar']/input[@type='text'] search-panel-button css #search-bar a menu-item-* css #menu li a = Main section = @on * menu: height 70px @on mobile login-button: width 100px @on mobile, desktop menu: height 300 px @set commonHeaderMargin 10 to 20px contentMargin ~ 20px # Approximate = Header = header-icon: inside header ${commonHeaderMargin} top left textfield: near button 5 to 15px left # By top edge menu-item-1: aligned horizontally top menu-item-2 # iframe中定义spec @objects banner-frame css iframe#banner = Main section = banner-frame: component frame banner.spec # color scheme login-form: color-scheme 10% white, 4 to 5 % black, < 30% #f845b7 # image menu-item-1: image file imgs/menu-item-1.png, error 4%, tolerance 80 # 循环 = Main section = @forEach [menu-item-*] as itemName, prev as previousItem ${itemName}: right-of ${previousItem} 10px @for [ 1, 2 ] as index menu-item-${index}: above menu-item-${index + 2} 0 to 5px

完整的例子



Check the md5sum for the whole folder

Sometimes we want to check if the specified svn tag code is the same as the trunk code. It is impossible for us to md5 by zip the whole folder, because after the zip, timestamp will involve in the zip file.
We can use find,grep,md5 together to solve this problem.
Here is the script i use to md5sum the whole folder.

find ./ -type f -name *.java |grep -v .svn|xargs md5  |md5

you should be caution that you should not check the parents folder name, or the md5sum will be differernt.
For example, if you hava tag folder name 20150923, and trunk name trunk. You have to use the script seperately for two folers.

cd trunk
find ./ -type f -name *.java |grep -v .svn|xargs md5  |md5
cd 20150923
find ./ -type f -name *.java |grep -v .svn|xargs md5  |md5

and compare each one.

前端自动化测试基础-断言篇:chai和chai插件的用法

chai

概念

测试技术的断言框架。

特点

  • 支持多种BDD/TDD断言语法
    • BDD:should
    • BDD:expect
    • TDD:assert
  • 可用在browser端和node端。
  • 可以和很多测试框架结合例如mocha jasmine等进行单元和UI测试。

安装

    npm install chai

用法

browser端

  <script src="//cdn.bootcss.com/chai/3.4.0/chai.js"></script>
  <script>
    //expect为全局的函数
     expect(foo).to.not.equal('bar');
  </script>

node端

var chai = require('chai'),
    expect = chai.expect;
chai.should();

expect用法


//------------------ 连接词用法 ----------------- //not用法 expect().not.to. expect(foo).to.not.equal('bar'); expect(goodFn).to.not.throw(Error); expect({ foo: 'baz' }).to.have.property('foo').and.not.equal('bar'); //deep用法 expect(foo).to.deep. 通常和equal连用,判断object的相等需要用deep expect(foo).to.deep.equal({ bar: 'baz' }); //any用法 用在keys的判断上 expect({ foo: 1, bar: 2 }).to.have.any.keys('foo', 'baz'); // all用法 用在keys的判断上 expect(foo).to.have.all.keys('bar', 'baz'); expect({ foo: 1, bar: 2 }).to.have.all.keys(['bar', 'foo']); //a 判断typeof 或者 language chain // typeof expect('test').to.be.a('string'); expect({ foo: 'bar' }).to.be.an('object'); expect(null).to.be.a('null'); expect(undefined).to.be.an('undefined'); // language chain expect(foo).to.be.an.instanceof(Foo); //---------------------判断bool---------------------- //bool // 1 truthy expect('everthing').to.be.ok; expect(1).to.be.ok; expect(false).to.not.be.ok; expect(undefined).to.not.be.ok; expect(null).to.not.be.ok; //2 true expect(true).to.be.true; expect(1).to.not.be.true; //3 false expect(false).to.be.false; expect(0).to.not.be.false; //4 null expect(null).to.be.null; expect(undefined).not.to.be.null; // 5 undefined expect(undefined).to.be.undefined; expect(null).to.not.be.undefined; //6 exist var foo = 'hi' , bar = null , baz; expect(foo).to.exist; expect(bar).to.not.exist; expect(baz).to.not.exist; //7 expty expect([]).to.be.empty; expect('').to.be.empty; expect({}).to.be.empty; //------------------------判断函数参数--------------------------- // arguments function test () { expect(arguments).to.be.arguments; } //------------------------判断相等和大小关系-------------------------------- // equal if the deep flag is set, // attention: asserts that the target is deeply equal to value. expect('hello').to.equal('hello'); expect(42).to.equal(42); expect(1).to.not.equal(true); expect({ foo: 'bar' }).to.not.equal({ foo: 'bar' }); expect({ foo: 'bar' }).to.deep.equal({ foo: 'bar' }); // eql: 判断值等 expect({ foo: 'bar' }).to.eql({ foo: 'bar' }); expect([ 1, 2, 3 ]).to.eql([ 1, 2, 3 ]); //.above:大于 expect(10).to.be.above(5); expect('foo').to.have.length.above(2); expect([ 1, 2, 3 ]).to.have.length.above(2); //least 至少 expect('foo').to.have.length.of.at.least(2); expect([ 1, 2, 3 ]).to.have.length.of.at.least(3); //below 低于 expect(5).to.be.below(10); expect('foo').to.have.length.below(4); expect([ 1, 2, 3 ]).to.have.length.below(4); //most 最大为 expect(5).to.be.at.most(5); expect('foo').to.have.length.of.at.most(4); expect([ 1, 2, 3 ]).to.have.length.of.at.most(3); //.within(start, finish)在什么区间内 expect(7).to.be.within(5,10); expect('foo').to.have.length.within(2,4); expect([ 1, 2, 3 ]).to.have.length.within(2,4); //.closeTo(expected, delta) expect(1.5).to.be.closeTo(1, 0.5); //------------------正则--------------- //match(regexp) expect('foobar').to.match(/^foo/); //-----------------字符串------------- //string 判断含有某字符串 expect('foobar').to.have.string('bar'); //----------------throw--------------- var err = new ReferenceError('This is a bad function.'); var fn = function () { throw err; } expect(fn).to.throw(ReferenceError); expect(fn).to.throw(Error); expect(fn).to.throw(/bad function/); expect(fn).to.not.throw('good function'); expect(fn).to.throw(ReferenceError, /bad function/); expect(fn).to.throw(err); expect(fn).to.not.throw(new RangeError('Out of range.')); //------------------------object相关判断------------------------- //deep & property属性 expect(foo).to.deep.equal({ bar: 'baz' }); expect({ foo: { bar: { baz: 'quux' } } }).to.have.deep.property('foo.bar.baz', 'quux'); // typeof expect('test').to.be.a('string'); expect({ foo: 'bar' }).to.be.an('object'); expect(null).to.be.a('null'); expect(undefined).to.be.an('undefined'); // language chain expect(foo).to.be.an.instanceof(Foo); // include expect([1,2,3]).to.include(2); expect('foobar').to.contain('foo'); expect({ foo: 'bar', hello: 'universe' }).to.include.keys('foo'); // members 判断数组成员 expect([1, 2, 3]).to.include.members([3, 2]); expect([1, 2, 3]).to.not.include.members([3, 2, 8]); expect([4, 2]).to.have.members([2, 4]); expect([5, 2]).to.not.have.members([5, 2, 1]); expect([{ id: 1 }]).to.deep.include.members([{ id: 1 }]); //respondTo(method) 判断是否是原型方法 Klass.prototype.bar = function(){}; expect(Klass).to.respondTo('bar'); expect(obj).to.respondTo('bar'); Klass.baz = function(){}; expect(Klass).itself.to.respondTo('baz'); //itself和respondTo结合起来判断是否是原型链的方法还是自身的方法 function Foo() {} Foo.bar = function() {} Foo.prototype.baz = function() {} expect(Foo).itself.to.respondTo('bar'); expect(Foo).itself.not.to.respondTo('baz'); //change 判断函数是否改变了对象的属性值 var obj = { val: 10 }; var fn = function() { obj.val += 3 }; var noChangeFn = function() { return 'foo' + 'bar'; } expect(fn).to.change(obj, 'val'); expect(noChangFn).to.not.change(obj, 'val') //increase(function) 函数是否升高了属性值 var obj = { val: 10 }; var fn = function() { obj.val = 15 }; expect(fn).to.increase(obj, 'val'); //.decrease(function) 函数是否降低了属性值 var obj = { val: 10 }; var fn = function() { obj.val = 5 }; expect(fn).to.decrease(obj, 'val'); //keys.判断是否object含有某项属性 //Note, either any or all should be used in the assertion. If neither are used, the assertion is defaulted to all. expect({ foo: 1, bar: 2 }).to.have.any.keys('foo', 'baz'); expect({ foo: 1, bar: 2 }).to.have.any.keys('foo'); expect({ foo: 1, bar: 2 }).to.contain.any.keys('bar', 'baz'); expect({ foo: 1, bar: 2 }).to.contain.any.keys(['foo']); expect({ foo: 1, bar: 2 }).to.contain.any.keys({'foo': 6}); expect({ foo: 1, bar: 2 }).to.have.all.keys(['bar', 'foo']); expect({ foo: 1, bar: 2 }).to.have.all.keys({'bar': 6, 'foo', 7}); expect({ foo: 1, bar: 2, baz: 3 }).to.contain.all.keys(['bar', 'foo']); expect({ foo: 1, bar: 2, baz: 3 }).to.contain.all.keys([{'bar': 6}}]);

should用法

同chai的差别详情参考

var chai = require('chai');
chai.should();
    //语法: 基本是 expect().to.xx 相当于 ().should.xx ****
    foo.should.be.a('string'); //expect(foo).to.be.a('string');
    foo.should.equal('bar'); //expect(foo).to.equal('bar');
    //省略用法,见expect

注意:should在IE9下有问题

assert

assert为TDD用法,现在一般都是用基于BDD的测试,所以省略,详情请参考 Assert

chai as promise用法 **

  • 将promise和chai结合起来,用于在某种异步的条件下形成的断言判断
  • attention: Chai as Promised is only compatible with modern browsers (IE ≥9, Safari ≥6, no PhantomJS)
  • 具体用法:参见

doSomethingAsync().then( function (result) { result.should.equal("foo"); done(); }, function (err) { done(err); } ); //安装: npm install chai-as-promised //引用chai as promise后可以写作 should.eventually.xxx var chai = require("chai"); var chaiAsPromised = require("chai-as-promised"); chai.use(chaiAsPromised); var should = chai.should(); return doSomethingAsync().should.eventually.equal("foo"); //在ui测试中可以写作 return driver.getAttribute(input, 'type').should.eventually.equal(fieldModel.type); return promise.should.be.fulfilled; return promise.should.eventually.deep.equal("foo"); return promise.should.become("foo"); // same as `.eventually.deep.equal` return promise.should.be.rejected; return promise.should.be.rejectedWith(Error); // other variants of Chai's `throw` assertion work too. // 通过覆盖chaiAsPromised.transferPromiseness方法将assertion赋予then的链式调用功能 // 应用例子 wd.js中 chaiAsPromised.transferPromiseness = wd.transferPromiseness; chaiAsPromised.transferPromiseness = function (assertion, promise) { assertion.then = promise.then.bind(promise); // this is all you get by default assertion.finally = promise.finally.bind(promise); assertion.done = promise.done.bind(promise); };

sinon-chai用法 **

  • sinon-chai 用于对Sinon.JS中的spy, stub, and mocking framework进行断言
  • 具体用法,参见
  • API为:
Sinon.JS property/method Sinon–Chai assertion
called spy.should.have.been.called
callCount spy.should.have.callCount(n)
calledOnce spy.should.have.been.calledOnce
calledTwice spy.should.have.been.calledTwice
calledThrice spy.should.have.been.calledThrice
calledBefore spy1.should.have.been.calledBefore(spy2)
calledAfter spy1.should.have.been.calledAfter(spy2)
calledWithNew spy.should.have.been.calledWithNew
alwaysCalledWithNew spy.should.always.have.been.calledWithNew
calledOn spy.should.have.been.calledOn(context)
alwaysCalledOn spy.should.always.have.been.calledOn(context)
calledWith spy.should.have.been.calledWith(…args)
alwaysCalledWith spy.should.always.have.been.calledWith(…args)
calledWithExactly spy.should.have.been.calledWithExactly(…args)
alwaysCalledWithExactly spy.should.always.have.been.calledWithExactly(…args)
calledWithMatch spy.should.have.been.calledWithMatch(…args)
alwaysCalledWithMatch spy.should.always.have.been.calledWithMatch(…args)
returned spy.should.have.returned(returnVal)
alwaysReturned spy.should.have.always.returned(returnVal)
threw spy.should.have.thrown(errorObjOrErrorTypeStringOrNothing)
alwaysThrew spy.should.have.always.thrown(errorObjOrErrorTypeStringOrNothing)

//安装 npm install sinon-chai //用法 var chai = require("chai"); var sinonChai = require("sinon-chai"); chai.should(); chai.use(sinonChai); function hello(name, cb) { cb("hello " + name); } describe("hello", function () { it("should call callback with correct greeting", function () { var cb = sinon.spy(); hello("foo", cb); cb.should.have.been.calledWith("hello foo"); //if expect expect(cb).to.have.been.calledWith("hello foo"); }); });

chai和mocha结合的测试用例

DataNode RPC to NameNode timeout

在启动Standby NameNode的后经常发现DataNode端报socket异常,具体日志是

java.net.SocketTimeoutException: Call From dn to ns2 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/dn:58211 remote=ns2]; For more details see:  http://wiki.apache.org/hadoop/SocketTimeout
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/dn:58211 remote=ns2]

并且,timeout只发生在DN向Standby发送Heartbeat的时候,而跟Active的通讯都是正常的。
初步怀疑是Standby 拉取Edit导致的full gc问题,但是相应时间standby的gc日志是完全正常的。随后jstack看到所有的时间都花费在updateCountForQuota方法上。updateCountForQuota方法就是从root节点开始递归的更新所有quota。
这步操作在Standby每次拉取editlog之后都会执行,我在测试集群生成1亿个INodeDir以后就可以稳定出现这一情况。
针对这个问题,我发了一个jira,地址是HDFS-9143,随后社区有个duplicate的jira,地址是HDFS-6763
我的解决方案就是每次editlog tail结束的时候不去调用updateCountForQuota方法,transition to active的时候调用一次就好了。类似于6763 V1方案,而V2,V3方案则对ActiveNameNode启动时候updateCountForQuota操作进行了优化。
这个改进将会消除Standby与DN之间经常出现的SocketTimeout同时也会加快Standby启动时间。

YARN MRAppMaster与Scheduler流程说明(三)

接上一篇说明,继续app提交在ResouceManager中的状态机流转。
9.FairScheduler.allocate
allocate方法基本上是申请mrappmaster和mrappmaster启动后向ResourceManager申请container资源使用的。它首先检查app是否在已经注册到scheduler中,然后检查申请资源状态,释放掉相应资源,并加入到AppSchedulingInfo结构中,这些内容会在MRAppmaster的流程中更具体的介绍。随后是获取能够运行appmaster的container资源的方法,代码为:

  public synchronized ContainersAndNMTokensAllocation
      pullNewlyAllocatedContainersAndNMTokens() {
    List<Container> returnContainerList =
        new ArrayList<Container>(newlyAllocatedContainers.size());
    List<NMToken> nmTokens = new ArrayList<NMToken>();
    for (Iterator<RMContainer> i = newlyAllocatedContainers.iterator(); i
      .hasNext();) {
      RMContainer rmContainer = i.next();
      Container container = rmContainer.getContainer();
      try {
        // create container token and NMToken altogether.
        container.setContainerToken(rmContext.getContainerTokenSecretManager()
          .createContainerToken(container.getId(), container.getNodeId(),
            getUser(), container.getResource()));
        NMToken nmToken =
            rmContext.getNMTokenSecretManager().createAndGetNMToken(getUser(),
              getApplicationAttemptId(), container);
        if (nmToken != null) {
          nmTokens.add(nmToken);
        }
      } catch (IllegalArgumentException e) {
        // DNS might be down, skip returning this container.
        LOG.error("Error trying to assign container token and NM token to" +
            " an allocated container " + container.getId(), e);
        continue;
      }
      returnContainerList.add(container);
      i.remove();
      rmContainer.handle(new RMContainerEvent(rmContainer.getContainerId(),
        RMContainerEventType.ACQUIRED));
    }
    return new ContainersAndNMTokensAllocation(returnContainerList, nmTokens);
  }

这里需要说明的是由于YARN大部分操作都是通过Distpatcher进行异步操作,NodeManager向ResouceManager汇报心跳的时候会最后触发FairScheduler的NODE_UPDATE事件,NodeManager会汇报最新执行的,执行结束的Container,Scheduler会放到相应的结构中。然后会从父节点开始分配,直到叶子节点,分配的规则会在调度器部门专门写。最后选出合适的队列,并且从队列里面再根据fair规则分配,检查注册的App是否有合适的需要分配,App中的分配是根据优先级进行的,appmaster>reduce>map,然后再根据本地性找合适的container进行分配。分配成功后会将生成一个RMContainer对象,代表一个运行的container,并且加入到newlyAllocatedContainers结构中。

    // Create RMContainer
    RMContainer rmContainer = new RMContainerImpl(container, 
        getApplicationAttemptId(), node.getNodeID(),
        appSchedulingInfo.getUser(), rmContext);
 
    // Add it to allContainers list.
    newlyAllocatedContainers.add(rmContainer);
    liveContainers.put(container.getId(), rmContainer);  
        // Update consumption and track allocations
    appSchedulingInfo.allocate(type, node, priority, request, container);
    Resources.addTo(currentConsumption, container.getResource());
 
    // Inform the container
    rmContainer.handle(
        new RMContainerEvent(container.getId(), RMContainerEventType.START));

并发送RMContainerEventType.START到dispatcher进行处理。
回到allocate方法,将刚才说到的newlyAllcatedContainers取出,并封装ContainersAndNMTokensAllocation返回,相当于调度成功。

10.RMContainerEventType.START
状态机及处理代码为:

    // Transitions from NEW state
    .addTransition(RMContainerState.NEW, RMContainerState.ALLOCATED,
        RMContainerEventType.START, new ContainerStartedTransition())
 
 
 
   private static final class ContainerStartedTransition extends
      BaseTransition {
 
    @Override
    public void transition(RMContainerImpl container, RMContainerEvent event) {
      container.eventHandler.handle(new RMAppAttemptContainerAllocatedEvent(
          container.appAttemptId));
    }
  }

就是出发RMAppAttemptEventType.CONTAINER_ALLOCATED给dispatcher调用RMAppimpl进行处理。

11.RMAppAttemptEventType.CONTAINER_ALLOCATED
触发RMAppAttemptEventType.CONTAINER_ALLOCATED事件的方法很多,但是目前为止RMAppimpl状态为RMAppAttemptState.SCHEDULED,通过之前的状态就很容易找到状态机处理函数的入口为:

       // Transitions from SCHEDULED State
      .addTransition(RMAppAttemptState.SCHEDULED,
          EnumSet.of(RMAppAttemptState.ALLOCATED_SAVING,
            RMAppAttemptState.SCHEDULED),
          RMAppAttemptEventType.CONTAINER_ALLOCATED,
          new AMContainerAllocatedTransition())

处理函数其实很简单,就是从get第9步获取的container,并返回。如果获取失败重新调度。否则store这个attempt,触发RMStateStoreEventType.STORE_APP_ATTEMPT事件,代码如下:

  public synchronized void storeNewApplicationAttempt(RMAppAttempt appAttempt) {
    Credentials credentials = getCredentialsFromAppAttempt(appAttempt);
 
    ApplicationAttemptState attemptState =
        new ApplicationAttemptState(appAttempt.getAppAttemptId(),
          appAttempt.getMasterContainer(), credentials,
          appAttempt.getStartTime());
 
    dispatcher.getEventHandler().handle(
      new RMStateStoreAppAttemptEvent(attemptState));
  }

12.RMStateStoreEventType.STORE_APP_ATTEMPT
这个事件很简单,就是根据配置是否保存和保存在哪种介质中,最后触发了RMAppAttemptEventType.ATTEMPT_NEW_SAVED事件,并发送到Dispatcher中。

13.RMAppAttemptEventType.ATTEMPT_NEW_SAVED
这个事件回到RMAppattemptImpl中处理,最后触发AMLauncherEventType.LAUNCH事件。

14.AMLauncherEventType.LAUNCH
这个事件由ApplicationMasterLauncher来处理,ApplicationMasterLauncher是ResouceManger中的一个服务,负责向NodeManager发送请求执行AppMaster。
AMLauncherEventType.LAUNCH的处理函数为

  private void launch(RMAppAttempt application) {
    Runnable launcher = createRunnableLauncher(application, 
        AMLauncherEventType.LAUNCH);
    masterEvents.add(launcher);
  }

masterEvents是一个BlockingQueue,ApplicationMasterLauncher有个线程LauncherThread负责调用生成的Runnable。至于与NodeManager交互放到下一篇中介绍。