前端自动化测试基础-sinon篇章

sinon用途

  • 在测试领域 Test double是很重要的一个概念。Test double主要用在自动化测试领域,会使用简单的对象或者流程模拟对应的行为减少测试的复杂性。
  • 用于 JavaScript 的测试监视(spy)、桩(stub)和仿制(mock)功能。不依赖其他类库,兼容任何单元测试框架。

sinon spy(最常用)

  • test spy 是这样的一类函数,它可以记录自己被调用的情况,包括传入的参数、返回结果、this 指向和抛出的错误(如果有的话)。test spy 可以是一个匿名函数,也可以是对一个已有函数进行的封装。
  • 用于测试callback函数
  • 用于spy已知行为的方法
  • 创建spy
    //创建一个匿名的函数用于记录调用的参数、返回值、以及异常
    var spy = sinon.spy();
    var spy = sinon.spy(myFunc);
    //对对象的方法增加spy 用于替换原有方法的行为,可以通过调用object.method.restore()实现恢复设置
    var spy = sinon.spy(object, "method");
  • sinon API用法
    • 判断某函数调用了某些参数:spy.withArgs(arg1[, arg2, …]); e.g. assert(spy.withArgs(42).calledOnce);
    • 某函数调用的次数:spy.callCount
    • spy.called
    • spy.calledTwice spy.calledThrice
    • spy.firstCall spy.secondCall spy.thirdCall
    • 判断是否在另一个spy之前(后)被调用 spy.calledBefore(anotherSpy);spy.calledAfter(anotherSpy);
    • 至少有一次被某个参数调用,参数可以部分匹配:spy.calledWith(arg1, arg2, …);
    • 至少有一次抛出异常:spy.threw();
  • sinon spy实例
        //以backbone的Event单测为例
        var eventer = _.extend({}, Backbone.Events),
            spy = sinon.spy();

        // Set up the spy.
        eventer.on("foo", spy);
        expect(spy.called).to.be.false;

        // Fire event.
        eventer.trigger("foo", 42);

        // Check number of calls.
        expect(spy.calledOnce).to.be.true;
        expect(spy.callCount).to.equal(1);

        // Check calling arguments.
        expect(spy.firstCall.args[0]).to.equal(42);
        expect(spy.calledWith(42)).to.be.true;

sinon stub

  • stub(桩)其实是最抽象最难理解的,Test stubs是一类预编码行为的函数(也是一种 spy)。除了改变stub对象的行为之外,它还支持所有的 spy API。同spy一样,stubs 可以是匿名函数,或者包装已有函数。当使用 stub 包装一个已有函数时,原函数将不会被调用。
  • stub用于:
    • 在测试中控制一个方法的行为,以强制代码沿特定路径执行。例如测试错误处理时,可以强制一个方法抛出错误。
    • 当你希望阻止一个方法被直接调用时(可能是因为这个方法触发了干扰行为,例如 XHR 请求之类的)。
  • 创建stub

    • 创建一个匿名的 stub 函数。var stub = sinon.stub();
    • 使用一个 stub 函数替代 object.method。原函数可以通过调用 object.method.restore() (或 stub.restore())方法来还原。如果 object.method 不是一个函数,则会抛出一个异常来帮助你避免类型错误。var stub = sinon.stub(object, “method”);
    • 使用 func 来替换 object.method,并且被包装在一个 spy 中。object.method.restore() 可以恢复原方法。var stub = sinon.stub(object, “method”, func);
    • stub 该对象的所有方法。var stub = sinon.stub(obj);
  • stub API,详情
    • stub.withArgs(arg1[, arg2, …]);
    • stub.returns(obj);
    • stub.throws(); 例: var callback = sinon.stub(); callback.withArgs(1).throws(“TypeError”);
    • stub.yieldsTo(property, [arg1, arg2, …])
  • sinon stub实例
//basic usage
    var obj = {
      multiply: function (a, b) { return a * b; },
      error: function (msg) { throw new Error(msg); }
    };

    it("stubs multiply", function () {
      // Stub with a hard-coded return value.
      sinon.stub(obj, "multiply").returns(5);
      expect(obj.multiply(1, 2)).to.equal(5);
      obj.multiply.restore();

      // Stub with a function.
      sinon.stub(obj, "multiply", function (a, b) {
        return a + b;
      });
      expect(obj.multiply(1, 2)).to.equal(3);
      obj.multiply.restore();
    });

    it("stubs error", sinon.test(function () {
      this.stub(obj, "error");
      expect(obj.error).to.not.throw();
    }));
  });

//use yieldsTo
    it("stubs with yieldsTo", function () {
            var obj = {
                    async: function (opts) {
                        opts.success("a", "b");
                    }
                },
                spyObj = {
                    failure: sinon.spy(),
                    success: sinon.spy()
                };

            sinon.stub(obj, "async").yieldsTo("success", 1, 2);

            // Call on object with callback spies.
            obj.async(spyObj);

            expect(spyObj.failure).to.have.not.have.been.called;
            expect(spyObj.success)
                .to.have.been.calledOnce.and
                .to.have.been.calledWith(1, 2);
        });

##sinon mock

  • 用于给出expectation然后验证某个object的method是否是正确的
  • 同spy的区别,mock出的object收到了数据或是调用并没有真正执行,一切针对mock的调用都是假的。所以mock可以用来测试具有side effect的函数,这里的side effect泛指和外部对象有数据交互或者是调用,比如调用外部对象的方法、向server发送数据、和UI对象有交互、写日志等等。
  • API
    • 创建mock:var mock = sinon.mock(obj);
    • 给出expectation mock.expects(“method”);
    • 校验是否正确: mock.verify();
    • 重置:mock.restore();
  • sinon mock 例子

describe("Sinon.JS mocks", function () {
  // Object literal with two methods.
  var obj = {
    multiply: function (a, b) { return a * b; },
    error: function (msg) { throw new Error(msg); }
  };

  it("mocks multiply", function () {
    // Create the mock.
    var mock = sinon.mock(obj);

    // The multiply method is expected to be called:
    mock.expects("multiply")
      .atLeast(2)    // 2+ times,
      .atMost(4)     // no more than 4 times, and
      .withArgs(2);  // 2 was first arg on *all* calls.

    // Make 3 calls to `multiply()`.
    obj.multiply(2, 1);
    obj.multiply(2, 2);
    obj.multiply(2, 3);

    // Verify **all** of the previous expectations.
    mock.verify();

    // Restore the object.
    mock.restore();
  });

});

sinon Fake XMLHttpRequest/ Fake Server

  • fake server用法
    {
        setUp: function () {
            this.server = sinon.fakeServer.create();//创建server
        },

        tearDown: function () {
            this.server.restore();
        },

        "test should fetch comments from server" : function () {
            this.server.respondWith("GET", "/some/article/comments.json",
                [200, { "Content-Type": "application/json" },
                 '[{ "id": 12, "comment": "Hey there" }]']);

            var callback = sinon.spy();
            myLib.getCommentsFor("/some/article", callback);
            this.server.respond();

            sinon.assert.calledWith(callback, [{ id: 12, comment: "Hey there" }]);
        }
    }

  • sinon可以用作实现request的模拟,现在更多的使用supertest
    用于HTTP的测试
describe('GET /user', function(){
  it('user.name should be an case-insensitive match for "tobi"', function(done){
    request(app)
      .get('/user')
      .set('Accept', 'application/json')
      .expect(function(res) {
        res.body.id = 'some fixed id';
        res.body.name = res.body.name.toUpperCase();
      })
      .expect(200, {
        id: 'some fixed id',
        name: 'TOBI'
      }, done);
  });
});

sinon 测试代码

DataXceiver本地读异常bug说明(HDFS-11802)

现象描述

用户在读取文件的时候报三台DN都无法取得该文件对应的block,经过fsck检查后没有发现该文件有丢块现象,到对应的dn上去查看日志,发现三台机器已经都处于不可读状态,报错为

2015-11-25 00:01:55,999 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 10.39.5.160:50010:DataXceiverServer:
java.io.IOException: Xceiver count 4097 exceeds the limit of concurrent xcievers: 4096
    at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:137)
    at java.lang.Thread.run(Thread.java:745)

很明显,这是超过了dataxceiver设置的最大的线程数4096,正常情况下是不可能超过的,所以说明dn有线程泄露的bug或者其它问题。
再检查日志,发现了以下的日志

Exception in thread "Thread-19" java.lang.IllegalStateException: failed to remove c53ce04928d1baa854f5dc1bfc8d565b
    at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
    at org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.removeShm(ShortCircuitRegistry.java:115)
    at org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry$RegisteredShm.handle(ShortCircuitRegistry.java:102)
    at org.apache.hadoop.net.unix.DomainSocketWatcher.sendCallback(DomainSocketWatcher.java:371)
    at org.apache.hadoop.net.unix.DomainSocketWatcher.access$1000(DomainSocketWatcher.java:52)
    at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:511)
    at java.lang.Thread.run(Thread.java:745)

看到这个日志以后就大概知道了原因,是由于DomainSocketWatcher线程异常退出,导致本地读线程没有回收机制,占满了所有的dataxceiver slot导致的。

问题原因

DomainSocketWatcher线程负责对本地读线程建立的socket进行一些处理和清理等。出问题的代码为

<code>
      try {
        while (true) {
              doSomecleanup.......
        }
      } catch (InterruptedException e) {
        LOG.info(toString() + " terminating on InterruptedException");
      } catch (IOException e) {
        LOG.error(toString() + " terminating on IOException", e);
      } finally {
        lock.lock();
        try {
          kick(); // allow the handler for notificationSockets[0] to read a byte
          Iterator<Entry> iter = entries.values().iterator();
          while(iter.hasNext()) {
            sendCallback("close", iter, fdSet);
          }
          entries.clear();
          fdSet.close();
        } finally {
          lock.unlock();
        }
      }
</code>

正常情况下代码不会走入到finally,而是一直在while中loop。而报出的异常则是remove一个共享内存对象的时候失败,而导致的运行时异常。
经过jira查询和我们集群中机器日志查看,发现原因为在于Client向DataNode申请本地读时候,DataNode建立共享内存对象以及File Descriptor出现异常,导致分配失败,日志为

2015-11-06 04:52:41,080 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: cliID: DFSClient_attempt_1435099124107_5925361_m_000028_0_1777694543_1, src: 127.0.0.1, dest: 127.0.0.1, op: REQUEST_SHORT_CIRCUIT_SHM, shmId: n/a, srvID: 01f352c6-4e63-4158-8ead-3e8146103b6f, success: false

而在DataXceiver的requestShortCircuitShm代码中,如果失败则close连接

      if ((!success) && (peer == null)) {
        // If we failed to pass the shared memory segment to the client,
        // close the UNIX domain socket now.  This will trigger the 
        // DomainSocketWatcher callback, cleaning up the segment.
        IOUtils.cleanup(null, sock);
      }

但是,所有close操作都是通过回调DomainSocketWatcher来做的,这样,当DomainSocketWatcher再次close的时候内存中的共享内存对象由于已经close被释放,而报runtimeerror,这样DomainSocketWatcher线程异常退出,本地读没有清理线程,慢慢占满了slot,最后导致了DataNode不可用。

解决方法

DataXceiver不负责close连接,而只是负责将与client连接shutdown,使client能够快速反应读异常,同时增加更多的catch,如果后续还有异常能够找到原因。

YARN MRAppMaster启动流程(一)

从这篇起,开始介绍MRAppmaster与NodeManager交互启动,以及申请资源等内容。

1.接YARN MRAppMaster与Scheduler流程说明,ApplicationMasterLauncher启动一个AMLaucher线程,用于启动MRAppmaster。首先是通过RPC连接,interface是ContainerManagementProtocol,负责ApplicationMaster向NodeManager通信,然后封装所有的运行环境,参数等到context中,随后就是通过RPC向NodeManager发送startContainer命令。

    StartContainersResponse response =
        containerMgrProxy.startContainers(allRequests);

至此,ResourceManager调度完成,下面的工作交给了NodeManager。

2.ContainerManagerImpl.startContainers
NodeManager是通过ContainerManagerImpl模块来处理ApplicationMasterLauncher的请求的,首先会对ugi,token进行验证,随后就启动container,获得启动的用户,执行的所有命令和conf,随后根据这些信息生成Application对象用来执行。

    Container container =
        new ContainerImpl(getConfig(), this.dispatcher, launchContext,
          credentials, metrics, containerTokenIdentifier);
    ApplicationId applicationID =
        containerId.getApplicationAttemptId().getApplicationId();
 
    Application application =
            new ApplicationImpl(dispatcher, user, applicationID, credentials, context);

随后通过NodeManager的dispatcher发送ApplicationEventType.INIT_APPLICATION事件。

3.ApplicationEventType.INIT_APPLICATION
ApplicationImpl处理ApplicationEventType.INIT_APPLICATION事件,通知logHandler处理

      // Inform the logAggregator
      app.dispatcher.getEventHandler().handle(
          new LogHandlerAppStartedEvent(app.appId, app.user,
              app.credentials, ContainerLogsRetentionPolicy.ALL_CONTAINERS,
              app.applicationACLs));

4.LogHandlerAppStartedEvent
我们集群开启了logAggregation,配置方法为在yarn-site.xml配置yarn.log-aggregation-enable为true。随后由LogAggregationService类进行处理,主要是初始化aggregator,代码为

  private void initApp(final ApplicationId appId, String user,
      Credentials credentials, ContainerLogsRetentionPolicy logRetentionPolicy,
      Map<ApplicationAccessType, String> appAcls) {
    ApplicationEvent eventResponse;
    try {
      verifyAndCreateRemoteLogDir(getConfig());
      initAppAggregator(appId, user, credentials, logRetentionPolicy, appAcls);
      eventResponse = new ApplicationEvent(appId,
          ApplicationEventType.APPLICATION_LOG_HANDLING_INITED);
    } catch (YarnRuntimeException e) {
      LOG.warn("Application failed to init aggregation", e);
      eventResponse = new ApplicationEvent(appId,
          ApplicationEventType.APPLICATION_LOG_HANDLING_FAILED);
    }
    this.dispatcher.getEventHandler().handle(eventResponse);
  }

随后向Dispatcher发送ApplicationEventType.APPLICATION_LOG_HANDLING_INITED事件。
5.ApplicationEventType.APPLICATION_LOG_HANDLING_INITED
由ApplicationImpl处理,代码为

  static class AppLogInitDoneTransition implements
      SingleArcTransition<ApplicationImpl, ApplicationEvent> {
    @Override
    public void transition(ApplicationImpl app, ApplicationEvent event) {
      app.dispatcher.getEventHandler().handle(
          new ApplicationLocalizationEvent(
              LocalizationEventType.INIT_APPLICATION_RESOURCES, app));
    }
  }

最后通过dispatcher发送ApplicationLocalizationEvent事件。
6. LocalizationEventType.INIT_APPLICATION_RESOURCES
在ContainerManagerImpl中注册了ApplicationLocalizationEvent的处理方法为ResourceLocalizationService,代码是

dispatcher.register(LocalizationEventType.class, rsrcLocalizationSrvc);

这时候开始了MRAppmaster本地化的过程,所谓本地化就是把执行需要的所有资源,环境从集群中下载到执行MRAppmaster的NodeManager中的相应目录中。关于这部分的内容,我推荐看hortonwokrs的这篇文章来仔细了解。
最后发送ApplicationInitedEvent事件到ApplicationImpl处理。到这一步,application初始化完毕,表示application在这台机器上运行,建立了相应的内存结构,下一步开始了初始化container。
7.ApplicationInitedEvent.APPLICATION_INITED
直接发送ContainerEventType.INIT_CONTAINER到distapcher。
8.ContainerEventType.INIT_CONTAINER
ContainerImpl来处理ContainerEventType事件,开始本地化过程。调用了RequestResourcesTransition函数来处理。这里会对所有的资源,根据PUBLIC,PRIVATE,APPLICATION分别处理放到 Map<LocalResourceVisibility, Collection> req 中,发送LocalizationEventType.INIT_CONTAINER_RESOURCES事件。
9.LocalizationEventType.INIT_CONTAINER_RESOURCES
对于7中不同类型的资源,转发给相应的tracker进行处理。处理事件为ResourceEventType.REQUEST
10.ResourceEventType.REQUEST
LocalizedResource来处理这个事件。开始下载过程。代码为

  private static class FetchResourceTransition extends ResourceTransition {
    @Override
    public void transition(LocalizedResource rsrc, ResourceEvent event) {
      ResourceRequestEvent req = (ResourceRequestEvent) event;
      LocalizerContext ctxt = req.getContext();
      ContainerId container = ctxt.getContainerId();
      rsrc.ref.add(container);
      rsrc.dispatcher.getEventHandler().handle(
          new LocalizerResourceRequestEvent(rsrc, req.getVisibility(), ctxt, 
              req.getLocalResourceRequest().getPattern()));
    }
  }

11.LocalizerResourceRequestEvent
由ResourceLocalizationService来处理,PUBLIC的资源由PublicLocalizer下载,PRIVATE和APPLICATION的资源由LocalizerRunner来处理。
PublicLocalizer内部有个queue为pending,做了同步,通过线程不断的拉文件系统的数据,通过FSDowload类来做。代码为

              pending.put(queue.submit(new FSDownload(lfs, null, conf,
                  publicDirDestPath, resource, request.getContext().getStatCache())),
                  request);

PRIVATE和APPLICATIOn则是通过LocalizerRunner线程不断拉取,最后有两种方式,一种是DefaultContainerExecutor一种是LinuxContainerExecutor(cgroup用来做资源隔离)来做,看一下默认的Default

//    ContainerLocalizer localizer =
//        new ContainerLocalizer(lfs, user, appId, locId, getPaths(localDirs),
//            RecordFactoryProvider.getRecordFactory(getConf()));
 
    createUserLocalDirs(localDirs, user);
    createUserCacheDirs(localDirs, user);
    createAppDirs(localDirs, user, appId);
    createAppLogDirs(appId, logDirs);
 
 
    //Path appStorageDir = getFirstApplicationDir(localDirs, user, appId);
    // randomly choose the local directory
    Path appStorageDir = getWorkingDir(localDirs, user, appId);
 
    String tokenFn = String.format(ContainerLocalizer.TOKEN_FILE_NAME_FMT, locId);
    Path tokenDst = new Path(appStorageDir, tokenFn);
    lfs.util().copy(nmPrivateContainerTokensPath, tokenDst);
    LOG.info("Copying from " + nmPrivateContainerTokensPath + " to " + tokenDst);
//    lfs.setWorkingDirectory(appStorageDir);
//    LOG.info("CWD set to " + appStorageDir + " = " + lfs.getWorkingDirectory());
    // TODO: DO it over RPC for maintaining similarity?
    FileContext localizerFc = FileContext.getFileContext(
        lfs.getDefaultFileSystem(), getConf());
    localizerFc.setUMask(lfs.getUMask());
    localizerFc.setWorkingDirectory(appStorageDir);
    LOG.info("Localizer CWD set to " + appStorageDir + " = " 
        + localizerFc.getWorkingDirectory());
    ContainerLocalizer localizer =
        new ContainerLocalizer(localizerFc, user, appId, locId, 
            getPaths(localDirs), RecordFactoryProvider.getRecordFactory(getConf()));
 
    localizer.runLocalization(nmAddr);

重要的是ContainerLocalizer,在runLocalization代码为

     exec = createDownloadThreadPool();
     CompletionService<Path> ecs = createCompletionService(exec);
     localizeFiles(nodeManager, ecs, ugi);

用线程池来下载集群中的资源到本地对应目录,并设置相应的权限。而localizeFiles则会周期性的向ResourceLocalizationService汇报状态,

         try {
            nodemanager.heartbeat(status);
          } catch (YarnException e) { }
          return;
        }
        cs.poll(1000, TimeUnit.MILLISECONDS);

在ResourceLocalizationService对应代码为

  @Override
  public LocalizerHeartbeatResponse heartbeat(LocalizerStatus status) {
    return localizerTracker.processHeartbeat(status);
  }

如果本地化全部完成,在processHeartbeat后会有状态机的改变

     case FETCH_SUCCESS:
            // notify resource
            try {
            getLocalResourcesTracker(req.getVisibility(), user, applicationId)
              .handle(
                new ResourceLocalizedEvent(req, ConverterUtils
                  .getPathFromYarnURL(stat.getLocalPath()), stat.getLocalSize()));

至此,资源本地化完成,container等待执行,此时event为ResourceLocalizedEvent。

响应式设计的自动化测试

响应式设计介绍

响应式设计(RWD)从2010年开始就逐渐进入人们的视线,虽然由于网速和网络的制约,目前国内都是采用针对pc和移动分别开发站点的策略,例如淘宝网的首页在pc端网页 ,在移动端的网页为网页,使用的是基于REM的布局设计。但是没人能否认响应式设计的重要性和简便性。响应式设计其实简单来说就是利用media query针对不同的设备和分辨率采用不同的css样式,用以达到网站在各个设备上的兼容性,再结合“移动优先”的策略,使得响应式设计更加的具有优势。如下图所示,就是一个简单的购物网站在不同设备上的响应式设计。

一个响应式设计的实现

下面就举一个简单的响应式网站的例子
代码见f2e-testing

  • 首页welcome页面
  • 登陆页面
  • notes列表页面
  • 新建note页面



公共头尾和菜单的响应式实现

这个属于很经典的响应式菜单和标题设计:
– 首页中如果是mobile或者ipad 则只显示标题的主要部分
– 菜单中如果是大屏,则一行显示菜单,mobile下用两行的菜单
– css完整实现:参见


<div id="header"> <div class="middle-wrapper"> <img id="header-logo" src="../src/assets/header-icon.png"> <h1> Sample Website <span class="not-on-mobile not-on-tablet">for Galen Framework</span></h1> </div> </div>
#menu {
    background: url("images/menu-background.png");
    margin: 0;
    color: white;
}
#menu ul {
    margin: 0;
    padding: 0;
    list-style: none;
}
#menu li {
    display: inline-block;
}
#menu ul:after {
    clear: both;
}
#menu li a {
    min-width: 100px;
    font-size: 1.2em;
    color: white;
    padding: 20px;
    display: inline-block;
}

@media (max-width: 500px) {
    .not-on-mobile {
        display: none;
    }

    #menu {
        width: 100%;
    }
    #menu li {
        width: 49%;
    }
    #menu li a {
        width: 100%;
    }
}
@media (max-width: 800px) {
    .not-on-tablet {
        display: none;
    }
}

welcome页面

welcome页面:使用的是bootstrap的jumbotron的布局,这款响应式布局
主要用在simple marketing or informational website。它具有一个通知的大型“布告栏”(jumbotron)和三栏式布局。
主要的实现代码:

    <div class="middle-wrapper">
        <div id="content">
            <div id="welcome-page" class="jumbotron">
                <h1>Welcome to our test page!</h1>

                <p>This app is used as a playground for <a href="http://galenframework.com/">Galen Framework</a></p>

                <p>
                    <button class="btn btn-lg btn-primary button-login" type="button" onclick="App.showLoginPage();">
                        Login
                    </button>
                </p>
                <p>To log in this website use the email <b>testuser@example.com</b> and password <b>test123</b>
                </p></div>
        </div>
    </div>

.jumbotron { padding: 30px; margin-bottom: 30px; color: inherit; background-color: #eee; } .jumbotron h1, .jumbotron .h1 { color: inherit; } .jumbotron p { margin-bottom: 15px; font-size: 21px; font-weight: 200; } .container .jumbotron { border-radius: 6px; } .jumbotron .container { max-width: 100%; } //大于768px @media screen and (min-width: 768px) { .jumbotron { padding-top: 48px; padding-bottom: 48px; } .container .jumbotron { padding-right: 60px; padding-left: 60px; } .jumbotron h1, .jumbotron .h1 { font-size: 63px; } } @media (max-width: 500px) { button { width: 100%; margin-top: 10px; } }

login登陆页面


@media (min-width: 501px) {
    .dialog-panel {
        width: 400px;
        border: 1px solid #ccc;
        padding: 20px;
        margin: auto;
        border-radius: 10px;
        box-shadow: 1px 3px 3px #ddd;
    }
}

响应式兼容尺寸

  • 合理的使用viewport

<meta name="viewport" content="width=device-width, initial-scale=1"> <!--或者使用如下的 --> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  • 使用hack兼容低版本浏览器的media query

<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries --> <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> <!--[if lt IE 9]> <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script> <script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script> <![endif]-->
  • 参考bootstrap 3 它优先使用“移动优先”原则,详情:参考
    • Extra small devices ~ Phones (< 768px) col-xs-
    • Small devices ~ Tablets (>= 768px) col-sm-
    • Medium devices ~ Desktops (>= 992px) col-md-
    • Large devices ~ Desktops (>= 1200px) col-lg-

/*========== Mobile First Method ==========*/ /* RWD is – Desktop -> Tablet -> Mobile */ /* Mobile First RWD is – Mobile -> Tablet -> Desktop */ /* Custom, iPhone Retina */ @media only screen and (min-width : 320px) { } /* Extra Small Devices, Phones */ @media only screen and (min-width : 480px) { } /* Small Devices, Tablets */ @media only screen and (min-width : 768px) { } /* Medium Devices, Desktops */ @media only screen and (min-width : 992px) { } /* Large Devices, Wide Screens */ @media only screen and (min-width : 1200px) { } /*========== Non-Mobile First Method ==========*/ /* Large Devices, Wide Screens */ @media only screen and (max-width : 1200px) { } /* Medium Devices, Desktops */ @media only screen and (max-width : 992px) { } /* Small Devices, Tablets */ @media only screen and (max-width : 768px) { } /* Extra Small Devices, Phones */ @media only screen and (max-width : 480px) { } /* Custom, iPhone Retina */ @media only screen and (max-width : 320px) { }

响应式设计的自动化测试框架 – galenframework

介绍

  • 用于响应式设计的开源UI自动化测试框架
  • 测试spec “语义化友好”,通过位置信息准确定位各个元素的位置
  • 测试用例API兼容java和javascript
  • pc端和无线端多尺寸兼容,支持selenium appium saucelab browserstack多服务器测试
  • 可自定义输出的测试 html report

安装

  • 下载二进制代码
  • 执行 ./install.sh
  • galen -v 显示如下命令行 表明安装成功
Galen Framework
Version: 2.1.2
JavaScript executor: Rhino 1.7 release 5 2015 01 29

测试环境建立

  • 执行 galen config:生成config文件用于配置初始化文件,具体参数配置 详情参见
  • 文件结构
    • tests文件夹:用于装载测试脚本
      • init.js: 用于配置测试的设备和尺寸
      • pages文件夹: ui自动化测试的Page Object页面
      • login.page.test.js(默认是以.test.js后缀作为测试文件,如果有特殊要求可以在config文件中配置)
    • specs文件夹: 用于装载响应式设计的规则spec文件
      • common.spec文件:
      • loginPage.spec文件等等
    • config文件:配置文件
    • reports目录:用于生成自动化测试的html结果

构建测试服务

  • appium作为mobile的测试服务器,android真机测试的服务搭建,参考
  • selenium作为pc端的测试服务器
#server端:8002端口启动三星galaxy SIII设备的测试服务器;8001端口启动IPAD模拟器;启动chromepc端的测试服务器
node . -a 127.0.0.1 -p 8002 -U 4df752b06833bfd3 --browser-name Chrome --no-reset
node . -a 127.0.0.1 -p 8001 --command-timeout 50000  --no-reset
selenium-standalone start

#客户端:测试 并且测试完成后浏览器打开测试结果

Galen的命令行运行,参考

  • galen check:运行spec
  • galen test: 运行测试用例
  • galen dump:生成可视化spec-
galen test mytest01.test
    --htmlreport "htmlreport-dir"
    --testngreport "report/testng.xml"
    --jsonreport "jsonreport-dir"
    --parallel-tests 4

galen test tests/ --htmlreport reports   

galen check homepage.gspec
    --url "http://example.com"
    --size "640x480"
    --javascript "some.js"
    --include "mobile,all"
    --exclude "toexclude"
    --htmlreport "htmlreport-dir"
    --testngreport "report/testng.xml"
    --jsonreport "jsonreport-dir"

galen dump "specs/homepage.gspec"
    --url "http://galenframework.com"
    --size "1024x768"
    --export "dumps/homepage-dump" 
    --max-width "200" 
    --max-height "200"    

测试流程

createGridDriver建立对服务器的链接,并启动driver


var driver = createGridDriver('http://127.0.0.1:8001/wd/hub',{ desiredCapabilities: { browserName: 'Safari', 'platformVersion': '9.1', 'platformName': 'iOS', 'app': 'safari', deviceName:"iPad Air", size: '600x800', safariInitialUrl: 'about:blank' } }); driver.get("http://test.xxxxx.com");

checkLayout连接spec文件和.test.js测试文件

  • 编写测试脚本
  • 编写spec文件

检查spec文件是否符合预期

//定义test
test("Simplest test", function () {
    // here goes a test code
});

//[] spec中 @on的tag名称
checkLayout(driver, "specs/welcomePage.spec", ['desktop']);

使用 Page Object Model

PageObject在selenium中是常见的设计模式,它可以快速的将测试用例和测试主体相互分开,通过复用,减少代码;同时可以把测试过程变化的参数在统一的地方配置,减少改动的成本。关于 Page Object我会再开文介绍,这里只为大家介绍在galenframework中我们可以如何快捷的定义我们的PageObject,以登陆页为参考:

$page(pageName, primaryFields, [ secondaryFields ])

this.LoginPage = $page("Login page", {
  email: "input.email", // css locator
  password: "xpath: //input[@class='password']", // xpath locator
  submitButton: "id: submit", // id locator

  load: function () {
    this.open("http://example.com/login");
    return this.waitForIt();
  },
  loginAs: function (userName, password) {
    this.email.typeText(userName);
    this.password.typeText(password);
    this.submitButton.click();
  }
});
// now you can use it like this
var loginPage = new LoginPage(driver).load();
loginPage.loginAs("testuser@example.com", "password");

页面的webdriver操作函数

参考 GalenPage.js

  • 针对于$page这个对象
    • open 打开页面
    • waitForIt 等到primaryFields的元素都
    • wait({}).untilAll({}); 等
    • getAllLocators:把
    • findChild 定位元素
    • findChildren 批量定位元素
  • 针对pageElement(primaryFields还有secondaryFields中的元素)
    • attribute 获取属性
    • cssValue 获得css属性值
    • typeText input输入内容
    • click 点击按钮
    • clear 清空input
    • getText 获得输入的内容
    • hover
    • getWebElement
    • findChild
    • findChildren
    • isDisplayed 元素是否展现

操作并输出到report中

  • logged(text, callback)
  • loggedFunction(textExpression, callback)

//${_1} ${_2} 代表arguments loggedFunction("Log-in as ${_1} with password ${_2}", function (email, password) { this.emailTextfield.typeText(email); this.passwordTextfield.typeText(password); this.submitButton.click(); });

spec文件编写

spec文件是用于描述元素css之间的各种关系,符合语义化的要求,详情 参考

  • 定义Objects:@objects
  • tags和sections:= Main section =
  • 变量 @set
  • import其他的spec文件规则:@import header.spec
  • forEach Loop:循环
  • near – checks that object is located near another object
  • below – checks that an element is located below other object
  • above – checks that an element is located above other object
  • left-of and right-of – checks that an element is located above other object
  • inside – checks that object is located inside another object
  • width – checks the width of object
  • height – checks the height of object
  • aligned – checks horizontal or vertical alignment of object with other objects on page
  • text – checks the text that is visible on page
    • text is – checks that text is exactly as expected
    • text contains -checks element contains expected text
    • text starts – element should start with expected text
    • text ends – element should end with expected text
    • text matches – verifies that text matches Java Regular Expression
  • centered – checks that object is centered inside another object
  • absent – checks that object is either missing on page or is not visible
  • contains – checks that object visually contains other objects inside it
  • on – checks that object is visually located on other object
  • component – runs a subset of specs from another file within the given object context
  • color-scheme – checks the color distribution in the given object area


@objects search-panel id search-bar search-panel-input xpath //div[@id='search-bar']/input[@type='text'] search-panel-button css #search-bar a menu-item-* css #menu li a = Main section = @on * menu: height 70px @on mobile login-button: width 100px @on mobile, desktop menu: height 300 px @set commonHeaderMargin 10 to 20px contentMargin ~ 20px # Approximate = Header = header-icon: inside header ${commonHeaderMargin} top left textfield: near button 5 to 15px left # By top edge menu-item-1: aligned horizontally top menu-item-2 # iframe中定义spec @objects banner-frame css iframe#banner = Main section = banner-frame: component frame banner.spec # color scheme login-form: color-scheme 10% white, 4 to 5 % black, < 30% #f845b7 # image menu-item-1: image file imgs/menu-item-1.png, error 4%, tolerance 80 # 循环 = Main section = @forEach [menu-item-*] as itemName, prev as previousItem ${itemName}: right-of ${previousItem} 10px @for [ 1, 2 ] as index menu-item-${index}: above menu-item-${index + 2} 0 to 5px

完整的例子



Check the md5sum for the whole folder

Sometimes we want to check if the specified svn tag code is the same as the trunk code. It is impossible for us to md5 by zip the whole folder, because after the zip, timestamp will involve in the zip file.
We can use find,grep,md5 together to solve this problem.
Here is the script i use to md5sum the whole folder.

find ./ -type f -name *.java |grep -v .svn|xargs md5  |md5

you should be caution that you should not check the parents folder name, or the md5sum will be differernt.
For example, if you hava tag folder name 20150923, and trunk name trunk. You have to use the script seperately for two folers.

cd trunk
find ./ -type f -name *.java |grep -v .svn|xargs md5  |md5
cd 20150923
find ./ -type f -name *.java |grep -v .svn|xargs md5  |md5

and compare each one.

前端自动化测试基础-断言篇:chai和chai插件的用法

chai

概念

测试技术的断言框架。

特点

  • 支持多种BDD/TDD断言语法
    • BDD:should
    • BDD:expect
    • TDD:assert
  • 可用在browser端和node端。
  • 可以和很多测试框架结合例如mocha jasmine等进行单元和UI测试。

安装

    npm install chai

用法

browser端

  <script src="//cdn.bootcss.com/chai/3.4.0/chai.js"></script>
  <script>
    //expect为全局的函数
     expect(foo).to.not.equal('bar');
  </script>

node端

var chai = require('chai'),
    expect = chai.expect;
chai.should();

expect用法


//------------------ 连接词用法 ----------------- //not用法 expect().not.to. expect(foo).to.not.equal('bar'); expect(goodFn).to.not.throw(Error); expect({ foo: 'baz' }).to.have.property('foo').and.not.equal('bar'); //deep用法 expect(foo).to.deep. 通常和equal连用,判断object的相等需要用deep expect(foo).to.deep.equal({ bar: 'baz' }); //any用法 用在keys的判断上 expect({ foo: 1, bar: 2 }).to.have.any.keys('foo', 'baz'); // all用法 用在keys的判断上 expect(foo).to.have.all.keys('bar', 'baz'); expect({ foo: 1, bar: 2 }).to.have.all.keys(['bar', 'foo']); //a 判断typeof 或者 language chain // typeof expect('test').to.be.a('string'); expect({ foo: 'bar' }).to.be.an('object'); expect(null).to.be.a('null'); expect(undefined).to.be.an('undefined'); // language chain expect(foo).to.be.an.instanceof(Foo); //---------------------判断bool---------------------- //bool // 1 truthy expect('everthing').to.be.ok; expect(1).to.be.ok; expect(false).to.not.be.ok; expect(undefined).to.not.be.ok; expect(null).to.not.be.ok; //2 true expect(true).to.be.true; expect(1).to.not.be.true; //3 false expect(false).to.be.false; expect(0).to.not.be.false; //4 null expect(null).to.be.null; expect(undefined).not.to.be.null; // 5 undefined expect(undefined).to.be.undefined; expect(null).to.not.be.undefined; //6 exist var foo = 'hi' , bar = null , baz; expect(foo).to.exist; expect(bar).to.not.exist; expect(baz).to.not.exist; //7 expty expect([]).to.be.empty; expect('').to.be.empty; expect({}).to.be.empty; //------------------------判断函数参数--------------------------- // arguments function test () { expect(arguments).to.be.arguments; } //------------------------判断相等和大小关系-------------------------------- // equal if the deep flag is set, // attention: asserts that the target is deeply equal to value. expect('hello').to.equal('hello'); expect(42).to.equal(42); expect(1).to.not.equal(true); expect({ foo: 'bar' }).to.not.equal({ foo: 'bar' }); expect({ foo: 'bar' }).to.deep.equal({ foo: 'bar' }); // eql: 判断值等 expect({ foo: 'bar' }).to.eql({ foo: 'bar' }); expect([ 1, 2, 3 ]).to.eql([ 1, 2, 3 ]); //.above:大于 expect(10).to.be.above(5); expect('foo').to.have.length.above(2); expect([ 1, 2, 3 ]).to.have.length.above(2); //least 至少 expect('foo').to.have.length.of.at.least(2); expect([ 1, 2, 3 ]).to.have.length.of.at.least(3); //below 低于 expect(5).to.be.below(10); expect('foo').to.have.length.below(4); expect([ 1, 2, 3 ]).to.have.length.below(4); //most 最大为 expect(5).to.be.at.most(5); expect('foo').to.have.length.of.at.most(4); expect([ 1, 2, 3 ]).to.have.length.of.at.most(3); //.within(start, finish)在什么区间内 expect(7).to.be.within(5,10); expect('foo').to.have.length.within(2,4); expect([ 1, 2, 3 ]).to.have.length.within(2,4); //.closeTo(expected, delta) expect(1.5).to.be.closeTo(1, 0.5); //------------------正则--------------- //match(regexp) expect('foobar').to.match(/^foo/); //-----------------字符串------------- //string 判断含有某字符串 expect('foobar').to.have.string('bar'); //----------------throw--------------- var err = new ReferenceError('This is a bad function.'); var fn = function () { throw err; } expect(fn).to.throw(ReferenceError); expect(fn).to.throw(Error); expect(fn).to.throw(/bad function/); expect(fn).to.not.throw('good function'); expect(fn).to.throw(ReferenceError, /bad function/); expect(fn).to.throw(err); expect(fn).to.not.throw(new RangeError('Out of range.')); //------------------------object相关判断------------------------- //deep & property属性 expect(foo).to.deep.equal({ bar: 'baz' }); expect({ foo: { bar: { baz: 'quux' } } }).to.have.deep.property('foo.bar.baz', 'quux'); // typeof expect('test').to.be.a('string'); expect({ foo: 'bar' }).to.be.an('object'); expect(null).to.be.a('null'); expect(undefined).to.be.an('undefined'); // language chain expect(foo).to.be.an.instanceof(Foo); // include expect([1,2,3]).to.include(2); expect('foobar').to.contain('foo'); expect({ foo: 'bar', hello: 'universe' }).to.include.keys('foo'); // members 判断数组成员 expect([1, 2, 3]).to.include.members([3, 2]); expect([1, 2, 3]).to.not.include.members([3, 2, 8]); expect([4, 2]).to.have.members([2, 4]); expect([5, 2]).to.not.have.members([5, 2, 1]); expect([{ id: 1 }]).to.deep.include.members([{ id: 1 }]); //respondTo(method) 判断是否是原型方法 Klass.prototype.bar = function(){}; expect(Klass).to.respondTo('bar'); expect(obj).to.respondTo('bar'); Klass.baz = function(){}; expect(Klass).itself.to.respondTo('baz'); //itself和respondTo结合起来判断是否是原型链的方法还是自身的方法 function Foo() {} Foo.bar = function() {} Foo.prototype.baz = function() {} expect(Foo).itself.to.respondTo('bar'); expect(Foo).itself.not.to.respondTo('baz'); //change 判断函数是否改变了对象的属性值 var obj = { val: 10 }; var fn = function() { obj.val += 3 }; var noChangeFn = function() { return 'foo' + 'bar'; } expect(fn).to.change(obj, 'val'); expect(noChangFn).to.not.change(obj, 'val') //increase(function) 函数是否升高了属性值 var obj = { val: 10 }; var fn = function() { obj.val = 15 }; expect(fn).to.increase(obj, 'val'); //.decrease(function) 函数是否降低了属性值 var obj = { val: 10 }; var fn = function() { obj.val = 5 }; expect(fn).to.decrease(obj, 'val'); //keys.判断是否object含有某项属性 //Note, either any or all should be used in the assertion. If neither are used, the assertion is defaulted to all. expect({ foo: 1, bar: 2 }).to.have.any.keys('foo', 'baz'); expect({ foo: 1, bar: 2 }).to.have.any.keys('foo'); expect({ foo: 1, bar: 2 }).to.contain.any.keys('bar', 'baz'); expect({ foo: 1, bar: 2 }).to.contain.any.keys(['foo']); expect({ foo: 1, bar: 2 }).to.contain.any.keys({'foo': 6}); expect({ foo: 1, bar: 2 }).to.have.all.keys(['bar', 'foo']); expect({ foo: 1, bar: 2 }).to.have.all.keys({'bar': 6, 'foo', 7}); expect({ foo: 1, bar: 2, baz: 3 }).to.contain.all.keys(['bar', 'foo']); expect({ foo: 1, bar: 2, baz: 3 }).to.contain.all.keys([{'bar': 6}}]);

should用法

同chai的差别详情参考

var chai = require('chai');
chai.should();
    //语法: 基本是 expect().to.xx 相当于 ().should.xx ****
    foo.should.be.a('string'); //expect(foo).to.be.a('string');
    foo.should.equal('bar'); //expect(foo).to.equal('bar');
    //省略用法,见expect

注意:should在IE9下有问题

assert

assert为TDD用法,现在一般都是用基于BDD的测试,所以省略,详情请参考 Assert

chai as promise用法 **

  • 将promise和chai结合起来,用于在某种异步的条件下形成的断言判断
  • attention: Chai as Promised is only compatible with modern browsers (IE ≥9, Safari ≥6, no PhantomJS)
  • 具体用法:参见

doSomethingAsync().then( function (result) { result.should.equal("foo"); done(); }, function (err) { done(err); } ); //安装: npm install chai-as-promised //引用chai as promise后可以写作 should.eventually.xxx var chai = require("chai"); var chaiAsPromised = require("chai-as-promised"); chai.use(chaiAsPromised); var should = chai.should(); return doSomethingAsync().should.eventually.equal("foo"); //在ui测试中可以写作 return driver.getAttribute(input, 'type').should.eventually.equal(fieldModel.type); return promise.should.be.fulfilled; return promise.should.eventually.deep.equal("foo"); return promise.should.become("foo"); // same as `.eventually.deep.equal` return promise.should.be.rejected; return promise.should.be.rejectedWith(Error); // other variants of Chai's `throw` assertion work too. // 通过覆盖chaiAsPromised.transferPromiseness方法将assertion赋予then的链式调用功能 // 应用例子 wd.js中 chaiAsPromised.transferPromiseness = wd.transferPromiseness; chaiAsPromised.transferPromiseness = function (assertion, promise) { assertion.then = promise.then.bind(promise); // this is all you get by default assertion.finally = promise.finally.bind(promise); assertion.done = promise.done.bind(promise); };

sinon-chai用法 **

  • sinon-chai 用于对Sinon.JS中的spy, stub, and mocking framework进行断言
  • 具体用法,参见
  • API为:
Sinon.JS property/method Sinon–Chai assertion
called spy.should.have.been.called
callCount spy.should.have.callCount(n)
calledOnce spy.should.have.been.calledOnce
calledTwice spy.should.have.been.calledTwice
calledThrice spy.should.have.been.calledThrice
calledBefore spy1.should.have.been.calledBefore(spy2)
calledAfter spy1.should.have.been.calledAfter(spy2)
calledWithNew spy.should.have.been.calledWithNew
alwaysCalledWithNew spy.should.always.have.been.calledWithNew
calledOn spy.should.have.been.calledOn(context)
alwaysCalledOn spy.should.always.have.been.calledOn(context)
calledWith spy.should.have.been.calledWith(…args)
alwaysCalledWith spy.should.always.have.been.calledWith(…args)
calledWithExactly spy.should.have.been.calledWithExactly(…args)
alwaysCalledWithExactly spy.should.always.have.been.calledWithExactly(…args)
calledWithMatch spy.should.have.been.calledWithMatch(…args)
alwaysCalledWithMatch spy.should.always.have.been.calledWithMatch(…args)
returned spy.should.have.returned(returnVal)
alwaysReturned spy.should.have.always.returned(returnVal)
threw spy.should.have.thrown(errorObjOrErrorTypeStringOrNothing)
alwaysThrew spy.should.have.always.thrown(errorObjOrErrorTypeStringOrNothing)

//安装 npm install sinon-chai //用法 var chai = require("chai"); var sinonChai = require("sinon-chai"); chai.should(); chai.use(sinonChai); function hello(name, cb) { cb("hello " + name); } describe("hello", function () { it("should call callback with correct greeting", function () { var cb = sinon.spy(); hello("foo", cb); cb.should.have.been.calledWith("hello foo"); //if expect expect(cb).to.have.been.calledWith("hello foo"); }); });

chai和mocha结合的测试用例

DataNode RPC to NameNode timeout

在启动Standby NameNode的后经常发现DataNode端报socket异常,具体日志是

java.net.SocketTimeoutException: Call From dn to ns2 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/dn:58211 remote=ns2]; For more details see:  http://wiki.apache.org/hadoop/SocketTimeout
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/dn:58211 remote=ns2]

并且,timeout只发生在DN向Standby发送Heartbeat的时候,而跟Active的通讯都是正常的。
初步怀疑是Standby 拉取Edit导致的full gc问题,但是相应时间standby的gc日志是完全正常的。随后jstack看到所有的时间都花费在updateCountForQuota方法上。updateCountForQuota方法就是从root节点开始递归的更新所有quota。
这步操作在Standby每次拉取editlog之后都会执行,我在测试集群生成1亿个INodeDir以后就可以稳定出现这一情况。
针对这个问题,我发了一个jira,地址是HDFS-9143,随后社区有个duplicate的jira,地址是HDFS-6763
我的解决方案就是每次editlog tail结束的时候不去调用updateCountForQuota方法,transition to active的时候调用一次就好了。类似于6763 V1方案,而V2,V3方案则对ActiveNameNode启动时候updateCountForQuota操作进行了优化。
这个改进将会消除Standby与DN之间经常出现的SocketTimeout同时也会加快Standby启动时间。

YARN MRAppMaster与Scheduler流程说明(三)

接上一篇说明,继续app提交在ResouceManager中的状态机流转。
9.FairScheduler.allocate
allocate方法基本上是申请mrappmaster和mrappmaster启动后向ResourceManager申请container资源使用的。它首先检查app是否在已经注册到scheduler中,然后检查申请资源状态,释放掉相应资源,并加入到AppSchedulingInfo结构中,这些内容会在MRAppmaster的流程中更具体的介绍。随后是获取能够运行appmaster的container资源的方法,代码为:

  public synchronized ContainersAndNMTokensAllocation
      pullNewlyAllocatedContainersAndNMTokens() {
    List<Container> returnContainerList =
        new ArrayList<Container>(newlyAllocatedContainers.size());
    List<NMToken> nmTokens = new ArrayList<NMToken>();
    for (Iterator<RMContainer> i = newlyAllocatedContainers.iterator(); i
      .hasNext();) {
      RMContainer rmContainer = i.next();
      Container container = rmContainer.getContainer();
      try {
        // create container token and NMToken altogether.
        container.setContainerToken(rmContext.getContainerTokenSecretManager()
          .createContainerToken(container.getId(), container.getNodeId(),
            getUser(), container.getResource()));
        NMToken nmToken =
            rmContext.getNMTokenSecretManager().createAndGetNMToken(getUser(),
              getApplicationAttemptId(), container);
        if (nmToken != null) {
          nmTokens.add(nmToken);
        }
      } catch (IllegalArgumentException e) {
        // DNS might be down, skip returning this container.
        LOG.error("Error trying to assign container token and NM token to" +
            " an allocated container " + container.getId(), e);
        continue;
      }
      returnContainerList.add(container);
      i.remove();
      rmContainer.handle(new RMContainerEvent(rmContainer.getContainerId(),
        RMContainerEventType.ACQUIRED));
    }
    return new ContainersAndNMTokensAllocation(returnContainerList, nmTokens);
  }

这里需要说明的是由于YARN大部分操作都是通过Distpatcher进行异步操作,NodeManager向ResouceManager汇报心跳的时候会最后触发FairScheduler的NODE_UPDATE事件,NodeManager会汇报最新执行的,执行结束的Container,Scheduler会放到相应的结构中。然后会从父节点开始分配,直到叶子节点,分配的规则会在调度器部门专门写。最后选出合适的队列,并且从队列里面再根据fair规则分配,检查注册的App是否有合适的需要分配,App中的分配是根据优先级进行的,appmaster>reduce>map,然后再根据本地性找合适的container进行分配。分配成功后会将生成一个RMContainer对象,代表一个运行的container,并且加入到newlyAllocatedContainers结构中。

    // Create RMContainer
    RMContainer rmContainer = new RMContainerImpl(container, 
        getApplicationAttemptId(), node.getNodeID(),
        appSchedulingInfo.getUser(), rmContext);
 
    // Add it to allContainers list.
    newlyAllocatedContainers.add(rmContainer);
    liveContainers.put(container.getId(), rmContainer);  
        // Update consumption and track allocations
    appSchedulingInfo.allocate(type, node, priority, request, container);
    Resources.addTo(currentConsumption, container.getResource());
 
    // Inform the container
    rmContainer.handle(
        new RMContainerEvent(container.getId(), RMContainerEventType.START));

并发送RMContainerEventType.START到dispatcher进行处理。
回到allocate方法,将刚才说到的newlyAllcatedContainers取出,并封装ContainersAndNMTokensAllocation返回,相当于调度成功。

10.RMContainerEventType.START
状态机及处理代码为:

    // Transitions from NEW state
    .addTransition(RMContainerState.NEW, RMContainerState.ALLOCATED,
        RMContainerEventType.START, new ContainerStartedTransition())
 
 
 
   private static final class ContainerStartedTransition extends
      BaseTransition {
 
    @Override
    public void transition(RMContainerImpl container, RMContainerEvent event) {
      container.eventHandler.handle(new RMAppAttemptContainerAllocatedEvent(
          container.appAttemptId));
    }
  }

就是出发RMAppAttemptEventType.CONTAINER_ALLOCATED给dispatcher调用RMAppimpl进行处理。

11.RMAppAttemptEventType.CONTAINER_ALLOCATED
触发RMAppAttemptEventType.CONTAINER_ALLOCATED事件的方法很多,但是目前为止RMAppimpl状态为RMAppAttemptState.SCHEDULED,通过之前的状态就很容易找到状态机处理函数的入口为:

       // Transitions from SCHEDULED State
      .addTransition(RMAppAttemptState.SCHEDULED,
          EnumSet.of(RMAppAttemptState.ALLOCATED_SAVING,
            RMAppAttemptState.SCHEDULED),
          RMAppAttemptEventType.CONTAINER_ALLOCATED,
          new AMContainerAllocatedTransition())

处理函数其实很简单,就是从get第9步获取的container,并返回。如果获取失败重新调度。否则store这个attempt,触发RMStateStoreEventType.STORE_APP_ATTEMPT事件,代码如下:

  public synchronized void storeNewApplicationAttempt(RMAppAttempt appAttempt) {
    Credentials credentials = getCredentialsFromAppAttempt(appAttempt);
 
    ApplicationAttemptState attemptState =
        new ApplicationAttemptState(appAttempt.getAppAttemptId(),
          appAttempt.getMasterContainer(), credentials,
          appAttempt.getStartTime());
 
    dispatcher.getEventHandler().handle(
      new RMStateStoreAppAttemptEvent(attemptState));
  }

12.RMStateStoreEventType.STORE_APP_ATTEMPT
这个事件很简单,就是根据配置是否保存和保存在哪种介质中,最后触发了RMAppAttemptEventType.ATTEMPT_NEW_SAVED事件,并发送到Dispatcher中。

13.RMAppAttemptEventType.ATTEMPT_NEW_SAVED
这个事件回到RMAppattemptImpl中处理,最后触发AMLauncherEventType.LAUNCH事件。

14.AMLauncherEventType.LAUNCH
这个事件由ApplicationMasterLauncher来处理,ApplicationMasterLauncher是ResouceManger中的一个服务,负责向NodeManager发送请求执行AppMaster。
AMLauncherEventType.LAUNCH的处理函数为

  private void launch(RMAppAttempt application) {
    Runnable launcher = createRunnableLauncher(application, 
        AMLauncherEventType.LAUNCH);
    masterEvents.add(launcher);
  }

masterEvents是一个BlockingQueue,ApplicationMasterLauncher有个线程LauncherThread负责调用生成的Runnable。至于与NodeManager交互放到下一篇中介绍。

YARN MRAppMaster与Scheduler流程说明(二)

上一篇主要介绍了一下YARN的基本概念以及Client的提交流程,这篇文章将会从ResourceManager入手,详解一下提交的流程。

ResouceManger接收任务

ResourceManager通过ClientRMService,首先要解析ApplicationSubmissionContext对象,获得提交的id,同时得到用户和提交的组等相应信息。随后生成一个RMApp,并在ResourceManager中注册,RMApp是Job的抽象,每个Job有一个RMApp与其对应。如果一切检验没问题后,正式提交给ResouceManager。
此后进入了状态机转换的部分,将分阶段说明:
1.RMAppEventType.START

      this.rmContext.getDispatcher().getEventHandler()
        .handle(new RMAppEvent(applicationId, RMAppEventType.START));

可以看到事件类型是RMAppEvent,再到ResourceManager中找相应代码,看看异步dispatcher注册的handle类是哪个,代码如下:

      // Register event handler for RmAppEvents
      rmDispatcher.register(RMAppEventType.class,
          new ApplicationEventDispatcher(rmContext));

进入ApplicationEventDipatcher看一下代码为

  @Private
  public static final class ApplicationEventDispatcher implements
      EventHandler<RMAppEvent> {
 
    private final RMContext rmContext;
 
    public ApplicationEventDispatcher(RMContext rmContext) {
      this.rmContext = rmContext;
    }
 
    @Override
    public void handle(RMAppEvent event) {
      ApplicationId appID = event.getApplicationId();
      RMApp rmApp = this.rmContext.getRMApps().get(appID);
      if (rmApp != null) {
        try {
          rmApp.handle(event);
        } catch (Throwable t) {
          LOG.error("Error in handling event type " + event.getType()
              + " for application " + appID, t);
        }
      }
    }
  }

这样我们就知道去找RMApp的实现类,RMAppImpl去找状态机改变及hook函数。其实也可以用比较简单的方法,就是在hadoop yarn代码中用grep去定位到包含RMAppEventType.START的类,基本上就八九不离十了。当然知道原来是首先需要掌握的。
随后看一下RMAppImpl的代码,状态机部分:

    .addTransition(RMAppState.NEW, RMAppState.NEW_SAVING,
        RMAppEventType.START, new RMAppNewlySavingTransition())

这表示状态RMAppEventType.START会触发RMAppNewlySavingTransition的hook函数,同时将RMAppState由NEW转换为NEW_SAVING。
在RMAppNewlySavingTransition的transition函数中调用了storeNewApplication

  public synchronized void storeNewApplication(RMApp app) {
    ApplicationSubmissionContext context = app
                                            .getApplicationSubmissionContext();
    assert context instanceof ApplicationSubmissionContextPBImpl;
    ApplicationState appState =
        new ApplicationState(app.getSubmitTime(), app.getStartTime(), context,
          app.getUser());
    dispatcher.getEventHandler().handle(new RMStateStoreAppEvent(appState));
  }

至此,将新的Event重新发到dispatcher中进行处理。

2.RMStateStoreEventType.STORE_APP
RMStateStoreEventType由RMStateStore类来处理,相应的代码如下:

      if(isRecoveryEnabled) {
        recoveryEnabled = true;
        rmStore =  RMStateStoreFactory.getStore(conf);
      } else {
        recoveryEnabled = false;
        rmStore = new NullRMStateStore();
      }
 
      try {
        rmStore.init(conf);
        rmStore.setRMDispatcher(rmDispatcher);
        rmStore.setResourceManager(rm);
      } catch (Exception e) {
        // the Exception from stateStore.init() needs to be handled for
        // HA and we need to give up master status if we got fenced
        LOG.error("Failed to init state store", e);
        throw e;
      }

默认的我们都是不带recovery,重启后需要重跑,因为recovery读大量目录,时间较慢,可以看到RMStateStore注册dispatcher中。
RMStateStore的handle函数,将event推送给handleStoreEvent方法进行处理,代码如下:

      LOG.info("Storing info for app: " + appId);
      try {
        if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
          storeApplicationStateInternal(appId, appStateData);
          notifyDoneStoringApplication(appId, storedException);
        } else {
          assert event.getType().equals(RMStateStoreEventType.UPDATE_APP);
          updateApplicationStateInternal(appId, appStateData);
          notifyDoneUpdatingApplication(appId, storedException);
        }
      } catch (Exception e) {
        LOG.error("Error storing app: " + appId, e);
        notifyStoreOperationFailed(e);
      }

处理完后会执行notifyDoneStoringApplication方法

  private void notifyDoneStoringApplication(ApplicationId appId,
                                                  Exception storedException) {
    rmDispatcher.getEventHandler().handle(
        new RMAppNewSavedEvent(appId, storedException));
  }

又向dispatcher中推送了一个RMAppNewSavedEvent的事件。

3.RMAppEventType.APP_NEW_SAVED
同1的处理,状态机最后将event推送到RMAppImpl处理,代码如下:

   .addTransition(RMAppState.NEW_SAVING, RMAppState.SUBMITTED,
        RMAppEventType.APP_NEW_SAVED, new AddApplicationToSchedulerTransition())

处理方式调用AddApplicationToSchedulerTransition

      app.handler.handle(new AppAddedSchedulerEvent(app.applicationId,
        app.submissionContext.getQueue(), app.user));

在ResouceManager中注册了AppAddSchedulerEvent的父类SchdulerEventType

      schedulerDispatcher = createSchedulerEventDispatcher();
      addIfService(schedulerDispatcher);
      rmDispatcher.register(SchedulerEventType.class, schedulerDispatcher);

Scheduler有自己独立的线程去处理,而不去堵塞ResourceManager的处理线程。由于有多重Scheduler,我这里只介绍我们使用的FairScheduler的流程,其它代码查看基本类似。

4.SchedulerEventType.APP_ADDED
FairScheduler的handle方法对多个事件进行处理,APP_ADDED的处理如下

    case APP_ADDED:
      if (!(event instanceof AppAddedSchedulerEvent)) {
        throw new RuntimeException("Unexpected event type: " + event);
      }
      AppAddedSchedulerEvent appAddedEvent = (AppAddedSchedulerEvent) event;
      addApplication(appAddedEvent.getApplicationId(),
        appAddedEvent.getQueue(), appAddedEvent.getUser());
      break;

在addAplication方法中,要获得提交时候生成的RMApp对象,提交的用户以及queue的名称,把任务放到queue中用于后面的调度,调用的代码是assignToQueue,这里会对有效性就行验证。随后在FairScheduler中生成SchedulerApplication对象,代表该任务被调度器接收等待调度,并放到Map<ApplicationId, SchedulerApplication> applications结构中。最后还是通过dispatcher将event转发出去。

    rmContext.getDispatcher().getEventHandler()
        .handle(new RMAppEvent(applicationId, RMAppEventType.APP_ACCEPTED));

5.RMAppEventType.APP_ACCEPTED
还是通过RMAppImpl来处理APP_ACCEPTED事件。处理方法如下:

  private static final class StartAppAttemptTransition extends RMAppTransition {
    @Override
    public void transition(RMAppImpl app, RMAppEvent event) {
      app.createAndStartNewAttempt(false);
    };
  }

这一步就是生成一个attempt,每个app每次执行都生成一个新的attempt,并保存在RMApp的内存结构中,attempt可能由于各种原因导致失败,app就会重新启动一个attempt,默认是超过4次,app就fail掉了。最后通过handler将事件发出处理。

6.RMAppAttemptEventType.START
RMAppAttemptEventType.START由刚刚生成的RMAppAttemptImpl来处理,如下代码:

      .addTransition(RMAppAttemptState.NEW, RMAppAttemptState.SUBMITTED,
          RMAppAttemptEventType.START, new AttemptStartedTransition())

RMAppAttemptEventType.START触发状态由NEW转化为SUBMITTED,并执行AttemptStartedTransition方法。AttemptStartedTransition方法内需要注意的调用为:

      // Register with the ApplicationMasterService
      appAttempt.masterService
          .registerAppAttempt(appAttempt.applicationAttemptId);
 
      // Add the applicationAttempt to the scheduler and inform the scheduler
      // whether to transfer the state from previous attempt.
      appAttempt.eventHandler.handle(new AppAttemptAddedSchedulerEvent(
        appAttempt.applicationAttemptId, transferStateFromPreviousAttempt));
    }

主要就是向applicationMaster注册,并且发起调度的事件,期望能够被Scheduler调度。

7.SchedulerEventType.APP_ATTEMPT_ADDED
FairScheduler处理APP_ATTEMPT_ADDED事件。

    SchedulerApplication application =
        applications.get(applicationAttemptId.getApplicationId());
    String user = application.getUser();
    FSLeafQueue queue = (FSLeafQueue) application.getQueue();
 
    FSSchedulerApp attempt =
        new FSSchedulerApp(applicationAttemptId, user,
            queue, new ActiveUsersManager(getRootQueueMetrics()),
            rmContext);
    if (transferStateFromPreviousAttempt) {
      attempt.transferStateFromPreviousAttempt(application
        .getCurrentAppAttempt());
    }
    application.setCurrentAppAttempt(attempt);
 
    boolean runnable = maxRunningEnforcer.canAppBeRunnable(queue, user);
 
    RMApp rmApp = rmContext.getRMApps().get(attempt.getApplicationId());
    queue.addApp(attempt, runnable,rmApp.getApplicationSubmissionContext().getPriority());
    if (runnable) {
      maxRunningEnforcer.trackRunnableApp(attempt);
    } else {
      maxRunningEnforcer.trackNonRunnableApp(attempt);
    }
 
    queue.getMetrics().submitAppAttempt(user);
    ClusterMetrics.getMetrics().addPendingApp(attempt.getApplicationId());
 
    LOG.info("Added Application Attempt " + applicationAttemptId
        + " to scheduler from user: " + user + " is isAttemptRecovering : " + isAttemptRecovering);
 
    if (isAttemptRecovering) {
      if (LOG.isDebugEnabled()) {
        LOG.debug(applicationAttemptId
            + " is recovering. Skipping notifying ATTEMPT_ADDED");
      }
    } else {
      rmContext.getDispatcher().getEventHandler().handle(
        new RMAppAttemptEvent(applicationAttemptId,
            RMAppAttemptEventType.ATTEMPT_ADDED));
    }

生成FSSchedulerApp,对应了RMApp中生成的RMAppAttempt。然后根据fair得原则,检查该任务是否可以执行,如果超出queue同时运行的任务数把状态置为目前不可运行,否则就是可运行状态。至此,app处于pending状态,根据队列的情况稍后可以调度还是等待转化为可以调度状态。最后通过dispatcher发送event,RMAppAttemptEventType.ATTEMPT_ADDED通过RMAppAttempt来处理。

8.RMAppAttemptEventType.ATTEMPT_ADDED
RMAppAttemptImpl处理ATTEMPT_ADDED事件,调用了SchedulerTransition代码,如下

    public RMAppAttemptState transition(RMAppAttemptImpl appAttempt,
        RMAppAttemptEvent event) {
      //Build blackList for AM
      Set<NodeId> failNodes = appAttempt.getFailMasterNodes();
      List<String> blackList = new ArrayList<String>(failNodes.size());
      for(NodeId node : failNodes) {
        blackList.add(node.toString());
      }
      if (!appAttempt.submissionContext.getUnmanagedAM()) {
        // Request a container for the AM.
        ResourceRequest request =
            BuilderUtils.newResourceRequest(
                AM_CONTAINER_PRIORITY, ResourceRequest.ANY, appAttempt
                    .getSubmissionContext().getResource(), 1);
 
        // SchedulerUtils.validateResourceRequests is not necessary because
        // AM resource has been checked when submission
        Allocation amContainerAllocation = appAttempt.scheduler.allocate(
            appAttempt.applicationAttemptId,
            Collections.singletonList(request), EMPTY_CONTAINER_RELEASE_LIST, blackList, null);
        if (amContainerAllocation != null
            && amContainerAllocation.getContainers() != null) {
          assert (amContainerAllocation.getContainers().size() == 0);
        }
        return RMAppAttemptState.SCHEDULED;
      } else {
        // save state and then go to LAUNCHED state
        appAttempt.storeAttempt();
        return RMAppAttemptState.LAUNCHED_UNMANAGED_SAVING;
      }
    }

这里向Scheduler申请一个container,用于执行appmaster,这个container的优先级是0,可以放到任意的机器上运行,同时将申请时候的内存使用量也写上,container个数为1。
提前说明一下,appmaster的优先级是0,reduce是10,map是20,数字越小越优先调度。随后调用FairScheduler的allocate方法,这是最重要的一个方法,包含了mrappmaster申请,mrappmaster向scheduler申请资源等,放到下一篇继续说明。

Appium的源码编译安装

Appium是现在比较活跃的开源自动化测试平台,因为更新速度很快,建议编译安装,了解其更多有意思的功能。
Appium支持ios android selendroid的自动化测试。在mac下配置ios环境还是相对简单的,但是android真机的配置就不是那么简单了,在此详细记录基于源码的编译安装。

准备工作 node

  • git clone https://github.com/appium/appium.git
  • 安装好node环境(brew安装最好)
  • 安装 mocha 和grunt-cli
npm install -g mocha
npm install -g grunt-cli

android真机配置

因为android虚拟器跑起来非常慢,如果不是专业的android的开发,安装跑andorid studio环境也没有必要
有对应的apk和sdk使用真机就能跑我们的测试脚本了。

准备工作:

  • 安装java jdk 配置JAVA_HOME
  • 安装android jdk,可以在线安装(国内速度超慢),所以快捷的方式是下载adt-bundle,解压后直接可用,下载地址
  • 配置ANDROID_HOME
  • 环境变量的配置代码见下方:
  • 执行环境检测 bin/appium-doctor.js –android 出现如下结果证明android环境配置成功
# ~/.bash_profile的配置内容
# 修改完之后source ~/.bash_profile生效

export ANDROID_HOME=/Users/zhangmeng/Documents/adt-bundle-mac-x86_64-20131030/sdk
export PATH=/Users/zhangmeng/Documents/adt-bundle-mac-x86_64-20131030/sdk/platform-tools:$PATH
export PATH=/Users/zhangmeng/Documents/adt-bundle-mac-x86_64-20131030/sdk/tools:$PATH
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home
export PATH=$JAVA_HOME/bin:$PATH

配置手机

  • 开启开发者选项,设置-{}开发者选项,如果没有找到,参考
  • 打开USB调试(如下图)
  • 部分手机需要在 连接USB的时候选用 MTP媒体模式才会生效
  • 在命令行执行如下指令,能够列出后(如果不行, 重新插拔一下usb,还可以尝试方法

adb kill-server
adb devices

其中list出来的就是手机的udid,用于后面的测试使用,如下图

执行初始化脚本

按照上面的步骤执行完成之后,运行命令./reset.sh –andorid –verbose即可。
在没有读这个reset.sh脚本的时候真的是被各种的环境搞的头晕脑胀,各种报错,包括:
基本都是有命令运行不通造成的,所以在这里大概介绍一下在appium reset android中的到底做了些什么,帮助大家理解这个启动脚本,以便配合自己的应用解决编译的问题,这个也是源码编译的好处之一,可以及时的解决更新服务。

  • android API 不匹配
  • Device chrome not configured yet
  • uninstall io.appium.android.ime卡住不再运行

reset.sh分析


reset_android() { echo "RESETTING ANDROID" require_java echo "* Configuring Android bootstrap" run_cmd rm -rf build/android_bootstrap run_cmd "$grunt" configAndroidBootstrap echo "* Building Android bootstrap" run_cmd "$grunt" buildAndroidBootstrap reset_unlock_apk reset_unicode_ime reset_settings_apk if $include_dev ; then reset_apidemos reset_toggle_test if $npmlink ; then link_appium_adb fi fi echo "* Setting Android config to Appium's version" run_cmd "$grunt" setConfigVer:android reset_chromedriver }
  • 配置Android bootstrap
    • 删除下build/android_bootstrap目录
    • 执行grunt configAndroidBootstrap:配置UiAutomation需要的编译文件 appium/lib/devices/android/bootstrap/build.xml project.properties local.properties
      • 生成AppiumBootstrap的编译文件:用于运行 android create uitest-project -n AppiumBootstrap -t android-19 -p xx/appium/lib/devices/android/bootstrap/
  • 编译 Android bootstrap
    • grunt buildAndroidBootstrap:使用ant编译AppiumBootstrap.jar,放置到appium/build/android_bootstrap/下
  • 编译apk文件(build目录下)
    • 编译 unlock apk: 唤醒和解锁andorid手机或是虚拟器详情
    • 编译 unicode ime apk: android对ASCII码的支持不好,所以会安装这个utf7的输入法,将sendKeys中的输入转为unicode识别的编码,详情
    • 编译 appium-settings apk:用于控制android系统 详情
  • 如果开启了测试模式 –dev参数
    • 编译sample-code下的app:ToggleTest apiDemos
  • 更新 appium-adb模块:运行./bin/npmlink.sh -l appium-adb
  • 更新appium的版本号
  • reset_chromedriver 详情参考

运行测试用例

  • node . -U 4df752b06833bfd3 (显示下面的提示证明Appium Server能够正常启动)
  • 详细的运行参数参考
  • 运行测试用例 : mocha wd-android-helloworld.js (wd.js
  • 其中支持原生的browser、chrome、还有apk的测试

var wd = require("wd");
var driver = wd.promiseChainRemote({
    host: 'localhost',
    port: 4723
});

driver
    .init({
        browserName: 'Chrome',//Chrome or Browser(原生,默认主页是google建议最好翻墙不然卡住)
        platformName: 'Android',
        platformVersion: '4.4.4',
        deviceName: 'Android Emulator'
      //,app: '/Users/zhangmeng/Downloads/com.taobao.taobao-5.3.1-121.apk' //如果选择测试app的内容 browserName设置为'';
      //执行app后会把对应的apk安装到真机中
    })
    .get('http://www.baidu.com')
    .sleep(5000)
    .title().then(function (title){
        console.log('this is the website title', title)
    })
    .quit()
    .done();

ios 虚拟器配置

配置和启动服务

$ git clone https://github.com/appium/appium.git
$ cd appium
$ ./reset.sh --ios --verbose
$ sudo ./bin/authorize-ios.js # for ios only 修改权限
$ node .

测试脚本

参见 safari-wd-search-test.js

参考

  • https://github.com/appium/appium/blob/master/docs/en/contributing-to-appium/appium-from-source.md
  • https://github.com/appium/appium/blob/master/docs/en/contributing-to-appium/grunt.md
  • http://university.utest.com/android-ui-testing-uiautomatorviewer-and-uiautomator/
  • http://developer.android.com/tools/help/shell.html