21.8.15

Log4J Configuration


Get the coordinates from maven central repo and add the dependency to your pom

    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>1.2.17</version>
    </dependency>
Create a file 'log4j.properties' and place it only under 'src/main/resources/' folder in your project, anywhere else, and it will give an error, like No Appenders found.....

This basic configuration would suffice in most cases, add it to the log4j.properties file.
 #---------------
# Root logger option
log4j.rootLogger=INFO, stdout, file

# Redirect log messages to console
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %C{1}.%M.%L: %m%n

# Redirect log messages to a log file, support file rolling.
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=E:\\ToDelete\\logger.log
log4j.appender.file.MaxFileSize=5MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %C{1}.%M.%L: %m%n
#---------------

NB:

  • To use the logger, just declare the class as follows.
    private final static Logger LOGGER = Logger.getLogger(BaseStep.class);

  • And you can start logging as follows.
LOGGER.info("Anything you want");

  • This would create logs in the format Class.Method.LineNo
  • For most part, we can set the logging level to INFO. DEBUG mode will cause log-blindness!
  • Also, if the appender file location [folder] does not exist, it will create it during run-time, provided it has the rights.
  • Although, it is recommended to declare this under each new class that you create, but if you declare it as public, in your base abstract class, then you would not have to declare it everywhere, and should be able to directly use it.


Maven - core ideas

Maven is a project management tool which encompasses a project object model, a set of standards, a project lifecycle, a dependency management system, and logic for executing plugin goals at the defined phases in a lifecycle.  Understanding how maven works is very critical for any developer.     


Maven reads the POM, then downloads the dependencies into the local repo, then executes the lifecycle, build phases and goals via the plugins, as per the specified build profile.

Philosophy
  • Maven suggests we follow (default) Convention over (custom) Configuration
  • Maven incorporates this concept by providing commonly followed default behavior for projects, without customization. The idea behind maven is about encouraging a set of standards, a common interface, a life-cycle, a standard repository format, a standard directory layout, etc.
  • The community needs to evolve beyond seeing technology as a zero-sum game between unfriendly competitors in a competition for users and developers.


Maven defaults –
  • Source code is assumed to be in ${basedir}/src/main/java
  • Resources are assumed to be in ${basedir}/src/main/resources
  • Tests are assumed to be in ${basedir}/src/test
  • A project is assumed to produce a JAR file. Maven assumes that you want the compile bytecode to ${basedir}/target/classes and then create a distributable JAR file in ${basedir}/target.
  • ~/.m2/settings.xml - A file containing user-specific configuration for authentication, repositories, and other information to customize the behavior of Maven.
  • ~/.m2/repository/ - This directory contains your local Maven repository. When you download a dependency from a remote Maven repository, Maven stores a copy of the dependency in your local repository.


POM –
  • POM is an XML representation of the project resources like dependencies, plugins, etc.
  • It is located in the root dir of the project.
  • POM specifies what needs to built, not how to build. How a project gets build is governed by the build phases and goals.
  • The modelVersion element sets what version of the POM model being used, and it corresponds to the the Maven version being used. Version 4.0.0 corresponds to Maven versions 2 and 3.
  • The groupId element corresponds to the name of the root java package of the project, but its not necessary that both be same, but again conventionally the pom groupId is same as the root package.
  • artifactID element contains the name of the java project. The JAR that gets eventually generated, uses the artifactID for the name.


Maven Build Process –
  • Maven Build Process consists of build lifecycle, phases and goals.
  • A build lifecycle consists of a pre defined sequence of phases, and each phase consists of a sequence of goals.
  • If we run a command to execute the lifecycle, then all the build phases in that lifecycle are executed.
  • If we run a command to execute the build phase, then all the build phases before it, in the pre defined sequence of build phases get executed too.
  • Sequence of Phases of Maven Build Lifecycle: validate >> compile >> test >> package >> verify >> install >> deploy
    • validate will check if the pom is correct
    • test will run the unit tests for classes
    • package will generate Jar/war as per POM file and will add the packaged jar or war to your target folder.
    • verify will run the integration tests
    • install will do all the things that package does, and then it will also add the packaged jar or war in your local .m2 repo
    • deploy will publish the jar to the remote/network repo

Maven Commands – 
  • To run the mvn command you have to pass the name of a build life cycle, phase or goal to it, which Maven then executes.
mvn install
This command executes the build phase called install. It executes all build phases before install in the build phase sequence, then executes the install phase, which builds the project and creates the packaged JAR file into the local Maven repository.
  • We can execute multiple build life cycles or phases by passing more than one argument to the mvn command.
mvn clean install
This command first executes the clean build life cycle, which removes compiled classes from the Maven output directory, and then it executes the install build phase. The target directory, under the project root dir, is created by Maven. It contains all the compiled classes, JAR files etc. produced by Maven. When executing the clean build phase, it is this target directory which gets cleaned.
  • You can also execute a Maven goal (a subpart of a build phase) by passing the build phase and goal name concatenated with a : in between, as parameter to the Maven command.
mvn dependency:resolve
     This would download all the jars for dependencies in the POM

mvn dependency:resolve -U
      U means force update of dependencies.

mvn help:effective-pom
This displays the entire pom for the project, containing even the inherited/hidden dependencies.

mvn integration-test --log-file log.log
This will run the integration test and output the result text in the log.log file in the root dir.

mvn archetype:generate
Here, 'mvn' is the Maven command and 'archetype:generate' is called a Maven goal. Also, 'archetype' is the identifier [ID] of a plugin and 'generate' is the identifier [ID] of the goal.


Plugin –
  • A Maven Plugin is a collection of one or more goals. A Plugin Contains Goals.
  • Examples of Maven plugins can be simple core plugins like the Jar plugin, which contains goals for creating JAR files, Compiler plugin, which contains goals for compiling source code and unit tests, Surefire plugin, which contains goals for executing unit tests and generating reports. 

Goal –
  • A Goal is a specific task that may be executed as a standalone goal or along with other goals as part of a larger build. A goal is a ‘unit of work’ in Maven.
  • Examples of goals include the compile goal in the Compiler plugin, which compiles all of the source code for a project, or the test goal of the Surefire plugin, which can execute unit tests.
  • The basic syntax for calling/executing a goal via a plugin is:
mvn pluginID:goalID
  • So basically, the idea is that there are different plugins and each plugin can do specialized set of tasks, and when we run the mvn command, we are basically telling mvn, to use a particular plugin, and with that plugin, accomplish a particular task [or 'goal'].
And, you should know various common plugins that would be used for automation - some plugins come bundled with maven core, and some might have to be added. And along with the maven plugins, you need to know various commands [goals] that each plugin can execute.


Lifecycle –
  • A Lifecycle is a default set of phases which get executed in the sequence defined for that life-cycle keyword, and in each phase various applicable goals are executed [various goals are 'bound' to phases].
Also, what is achieved via the life-cycle command can also be achieved by specifying the plugin goals in the same sequence as they would be executed via the life-cycle. But it is much easier to execute lifecycle phases than it is to specify explicit goals on the command line, and the common lifecycle allows every project that uses Maven to adhere to a well-defined set of standards.
  • Maven coordinates are often written using a colon as a delimiter in the following format: groupId:artifactId:packaging:version. In the pom.xml file for a project, its coordinates are represented as mavenbook:myapp:jar:1.0-SNAPSHOT
A project’s groupId:artifactId:version combination make that project unique.
  • The packaging format of a project is also an important component in the Maven coordinates, but it isn’t a part of a project’s unique identifier. A project with packaging 'jar' produces a JAR archive; a project with packaging 'war' produces a web application. Also, when you create a JAR for a project, dependencies are not bundled with the generated artifact; they are used only for compilation. When you use Maven to create a WAR or an EAR file, you can configure Maven to bundle dependencies with the generated artifact.
  • Maven has 3 built in lifecycles - default, clean and site. The default life cycle handles everything related to compiling and packaging your project. The clean life cycle handles everything related to removing temporary files from the output directory, including generated source files, compiled classes, previous JAR files etc. The site life cycle handles everything related to generating documentation for your project. You cannot execute the default life cycle directly. You have to specify a build phase or goal inside the default life cycle.

GroupId –

    • The group, company, team, organization, project, or other group. The convention for group identifiers is that they begin with the reverse domain name of the organization that creates the project.

ArtifactId –

    • A unique identifier under groupId that represents a single project.

Version –

    • A specific release of a project. Projects undergoing active development can use a special identifier that marks a version as a SNAPSHOT.


Dependency –
  • Dependency is an external JAR used in the project.
  • One of the most basic uses of maven is to manage all the external JARs used in the project, by adding them as dependencies in the POM.
If dependencies are not found in the local repo, then maven downloads them from the Central Repo and puts them in the local repo. [Local repo is just a folder containing all the jars, generally located in the .m2 folder under each user's profile folder]
  • Even if a maven dependency is not available in the central repo, we can still put the dependency [jar] in the local repo ourselves. To put the dependency manually, we need to adhere to the groupId, artifactID and version format while creating the directory structure.
            <dependency>
                        <groupId>org.seleniumhq.selenium</groupId>
                        <artifactId>selenium-java</artifactId>
                        <version>2.46.0</version>
            </dependency>
For the above dependency, the selenium jar needs to be located under MAVEN_REPOSITORY_ROOT/org/seleniumhq/selenium/selenium-java/2.46.0/ This can be done for jars which have a simple structure, but for jars with multiple sub dir it can be tricky.

Maven reads the pom file, then downloads the dependencies into the local repo, then executes the lifecycle, build phases and goals via the plugins, as per the specified build profile.

External Dependencies -


An external dependency in Maven is a dependency (JAR file) which is not located in a Maven repository (neiterh local, central or remote repository). The word "external" thus means external to the Maven repository system - not just external to the project. Most dependencies are external to the project, but few are external to the repository system (not located in a repository) - these can be proprietory jars which are usually not free/OS. There 3 options to use such jars in the project.

Option 1: We can directly add such jars in the IDE and run our tests via testNG. We can keep such jars in the src/main/resources/lib folder and add their references in the project build path. Then these jars can be committed in the VCS and as such would be present along with the source code of the project. As such we can now run our tests via TestNG without any issues. But we cannot run like this on CI because these jars are not present in the POM, hence, we have to use option 2.

Option 2: Keep these jars in the src/main/resources/lib folder in the project. Then add their dependencies in the POM with dummy groupId, artifactID and version. But the path to these jars should always be relative to the src dir. We configure an external dependency like below:
<dependency>
<groupId>mydependency</groupId>
<artifactId>mydependency</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/resources/lib/mydependency.jar</systemPath>
</dependency>

Option 3: Use the maven-install-plugin or the maven-external-dependency-plugin.
  • If a project A depends on project B, which in turn depends on project C, then we only have to include the dependency of project B in the pom of project A -  Maven takes care of child dependencies for project B [project C] implicitly.
Also, there is an idea of dependency 'scope' for a particular 'goal'. When a dependency has a scope of test, it will not be available to the compile goal of the Compiler plugin, it would be available only for the 'test'-esque goals. A test-scoped dependency is a dependency that is available on the classpath only during test compilation and test execution. If your project has war or ear packaging, a test-scoped dependency would not be included in the project’s output archive. To add a test-scoped dependency, add the dependency element to your project’s dependencies section, as shown in the following example:

 <dependency>
   <groupId>org.apache.commons</groupId>
   <artifactId>commons-io</artifactId>
   <version>1.3.2</version>
   <scope>test</scope>
 </dependency>


SureFire plugin to run TestNG suite-
  • Maven SureFire Plugin is a very widely used plugin to execute tests
  • You don’t have to do anything special to run a unit test; the test phase is a normal part of the Maven lifecycle. You run Maven tests whenever you run 'mvn package' or 'mvn install' commands. If you would like to run all the lifecycle phases up to and including the test phase, run 'mvn test' command.
  • When Maven encounters a build failure, its default behavior is to stop the current build. To continue building a project even when the Surefire plugin encounters failed test cases, you’ll need to set the testFailureIgnore configuration property of the Surefire plugin to true.
<build>
 <plugins>
  <plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
     <testFailureIgnore>true</testFailureIgnore>
    </configuration>
  </plugin>
 </plugins>
</build>

This property can also be set from the command line using the -D parameter:
mvn test -Dmaven.test.failure.ignore=true

Maven also provides for the ability to skip unit tests using the skip parameter of the Surefire plugin. To skip tests from the command line, simply add the maven.test.skip property to any goal:
mvn install -Dmaven.test.skip=true

Another way to configure Maven to skip unit tests is to add this configuration to your project’s pom.xml. To do this, you would add a plugin element to your build configuration.
<build>
 <plugins>
  <plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
     <skip>true</skip>
    </configuration>
  </plugin>
 </plugins>
</build>


  • To have Maven run our tests in parallel, we need to run the tests as 'methods' and not 'classes', along with the fork count being equal to the runner methods, without re-using the threads/forks.
  • If we run tests as 'classes', then they will not run in parallel. Basically, the way you define the execution config for TestNG, the same should be followed for SureFire also.
  • Maven and its plugins use the TestNG Annotations to invoke the methods/classes as tests via SureFire plugin.
  • Just add the below build tag in the pom before dependency tags to control how you build and run tests


<build>
<!-- Source directory configuration -->
<sourceDirectory>src</sourceDirectory>
<plugins>

<!-- Following plugin executes the TestNG tests -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M4</version>
<configuration>
<parallel>methods</parallel>
<forkCount>3</forkCount>
<!-- TestNG Suite Config file to use for test execution -->
<suiteXmlFiles>
<suiteXmlFile>./Config/testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>

<!-- Compiler plugin configures java version for compiling the code -->
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>


Passing TestNG XML at run-time via mvn -

  • Running TestNG via command line is very cumbersome [as it involves knowing and passing various parameters and even updating the ClassPath Env Var] hence, instead its much better to run the tests via Maven SureFire plugin. We should use TestNG XML just to specify the tests to be run.
  • We can even the testNG xml at run time via mvn command and then read them in the POM to run that suite, like below.
<suiteXmlFiles>
<suiteXmlFile>${suiteXmlFile}</suiteXmlFile>
                        <suiteXmlFile>./Config/testng.xml</suiteXmlFile> 
</suiteXmlFiles>
  • And call as below
mvn test -DsuiteXmlFile=/dirPath/fileName.xml
  • Or as
mvn test -Dsurefire.suiteXmlFile=/dirPath/fileName.xml
  • In a way, this also shows how we can pass any 'value' via mvn command and then have them read as '{parameters}' anywhere in the pom.xml
  • We can even specify multiple testNG XMLs and when we run the tests they get executed in the sequence they are added in the pom. The only issue with this is that none of the tests would run and execution would fail if even one of the XMLs is not specified/empty in case we just use the 'mvn test' command.
  •  Just remember that mvn commands are case sensitive so, ensure to use exact parameter names in '-D' coordinates.



Maven Settings File –
  • Maven has two settings files. In the settings files you can configure settings for Maven across all Maven POM files. The two settings files are located at:
The Maven installation directory: $M2_HOME/conf/settings.xml
The user's home directory: ${user.home}/.m2/settings.xml
  • Both files are optional. If both files are present, the values in the user home settings file overrides the values in the Maven installation settings file.
  • We can change the location of the local repository by setting the directory inside our Maven settings file. Maven settings file is also located in our user-home/.m2 directory and is called settings.xml. Here is how we can specify another location for the local repository:
<settings>
    <localRepository>
        d:\data\java\products\maven\repository
    </localRepository>
</settings>


Maven Central Repository -


Remote Repository –
  • Sometimes for some stupid reason or just to frustrate everyone companies decide to change the internal source repo and make the new repo just imposible to access or have limited connectivity via some type of servers. Hence, its a good idea to mention the exact repo that you are using directly in the pom because the repo in the pom overrides the repo mentioned in the local settings.xml.
  • A remote repository is a repository on a web server from which Maven can download dependencies, just like the central repository. A remote repository can be located anywhere on the internet, or inside a local network.
  • You can configure a remote repository in the POM file. Put the following XML elements right after the <dependencies> element:
<repositories>
   <repository>
       <id>automation.code</id>
       <url>http://test.automation.com/maven2/lib</url>
   </repository>
</repositories>


Build Profiles –
  • Maven build profiles enable you to build your project using different configurations. Instead of creating two separate POM files, you can just specify a profile with the different build configuration, and build your project with this build profile when needed.
  • Maven build profiles are specified inside the POM file, inside the profiles element. Each build profile is nested inside a profile element.
  • Build Profiles help build the same maven project in different ways, according to the end use. 
  • For example, the build profile for test and production environments would be different, again, the profile to be used for test automation would be different, and can have its own specifications as per execution environments [on local desktop or on CI box]. These different build profiles can be added to the pom and specified in the maven command.
 <profiles>
      <profile>
          <id>test</id>
          <activation>...</activation>
          <build>...</build>
          <modules>...</modules>
          <repositories>...</repositories>
          <pluginRepositories>...</pluginRepositories>
          <dependencies>...</dependencies>
          <reporting>...</reporting>
          <dependencyManagement>...</dependencyManagement>
          <distributionManagement>...</distributionManagement>
      </profile>


  </profiles>


<resources> tag in the POM - 
  • This feature allows you, for example, to use the variables defined in the POM, inside your .properties files. Just declare the directory, where your .properties are, in <resources> section, and you'll be able to use ${project.name}.
  • Refer - http://maven.apache.org/plugins/maven-resources-plugin/examples/filter.html
<resources>
        <resource>
            <directory>src/main/resources</directory>
            <filtering>true</filtering>
        </resource>
    </resources>
              
  • This is not essential though could be useful in some cases for passing Env variables via maven coordinates.

<pluginManagement> tag in the POM -
  • Difference between plugins and plugiManagement tags –
    • <pluginManagement/> defines the settings for plugins that will be inherited by modules in your build. This is great for cases where you have a parent pom file.
    •  <plugins/> is an actual invocation of the plugin. It may or may not be inherited from a <pluginManagement/>.
  • pluginManagement: is an element that is seen along side plugins. Plugin Management contains plugin elements in much the same way, except that rather than configuring plugin information for this particular project build, it is intended to configure project builds that inherit from this one. However, this only configures plugins that are actually referenced within the plugins element in the children. The children have every right to override pluginManagement definitions.
  • You don't need to have a <pluginManagement/> in your project, if it's not a parent POM.
  • This is not essential for the JB project, the CC POM does not have it.
  • Refer - http://stackoverflow.com/questions/10483180/maven-what-is-pluginmanagement/

Maven Folder Structure -

  • Resources folder - 
    • Usually the 'resources' folder in all maven projects is located under - ./src/main/resources
    • If for some reason you decide to keep your data files (like .xls) in this folder and if you have any of it opened while you are running the cmd 'mvn test' then the step would fail with the error that ab.xls is already open and being used by other process - even if that data file is not being used in the current cmd at all. So, try to avoid keeping data-files in the resources folder, if you have to keep many of them open while running tests


ANT -

  • Maven Ant is Deprecated.
  • Gradle = Maven + Ant != good for QA
  • Ant is just a build tool, and is very procedural, in the sense, everything has to be defined explicitly, as there are no defaults.




Maven command cheatsheet

All commands to be executed from the project root folder containing the pom.xml

mvn help:effective-pom
Displays a much larger POM which exposes the default settings of Maven.

mvn jar:jar
This packs up [creates a package] the entire project into a .JAR file, and places it under {basedir}/target folder, with a name like 'BDD-1.0-SNAPSHOT.jar'.
This can be used to zip-up the projects and move around.

mvn site
This will execute the site lifecycle phase. This lifecycle is concerned solely with processing site content under the src/site directories and generating reports. After this command executes, you should see a project web site in the target/site directory. Load target/site/index.html and you should see a basic shell of a project site. 
You can customize site reports by configuring which reports are included in a build via the pom.xml file. JBehave uses this feature to create those result reports, and the format of those reports is implemented via .FTL files.

mvn dependency:resolve
This will list down all the dependencies [jars, versions, etc], and their scope, including the implicit transient [child] dependencies.

mvn dependency:tree
This will list down all the dependencies in your pom, along with the transient dependency for each of those - which dependency in your pom has its own implicit child dependency.

mvn install assembly:assembly
In this the 'assembly' goal is executed by the 'assembly' plugin, after the Maven build reaches the 'install' lifecycle phase. This shows an example of commands using life-cycle phases and plugin goals.

mvn clean > clean.txt
This will run the clean command and output the result text in the clean.txt file in the root dir.



4.8.15

Pass and read Runtime Property Values via Maven command

Many a times when you implement CI with Maven & Jenkins or TeamCity, you might want to pass parameters to the test like the Browser and Env in which you want to run your tests.
This can be done in various ways, but a very simple and straightforward method is as follows -

Define the variables in your tests that you want to store the values of these run-time parameters, like below

static final String BROWSER_TYPE = System.getProperty("browserType");
static final String RUN_ENV = System.getProperty("runEnv");

These can be defined in the base Abstract class that you use to initialize global methods/fields, etc. And its a good idea to declare them static and final so that we can use them directly in our tests, and we dont overwrite these values.

System.getProperty - this is the method that actually gets you the value that you have passed as part of your maven command. You can get any property you want via this.

Your maven command can look like this -

    mvn integration-test -DbrowserType=chrome -DrunEnv=test3

You can have these parameters defined in any order, and can give any name to it, but the names passed as maven command, and those referred in your tests should be the same. What I refer here as parameters is technically called a property of the plugin.

In order to pass multiple values for the same parameters, use the following syntax -

mvn integration-test -DbrowserType=FireFox-DrunEnv=Test1 -DmetaFilters=+Regression +OnBoarding -skip -Manual -wip

Multiple values have been specified for the metaFilters property, to be used with JBehave Test Story-s and Scenarios.
This will only execute scenarios with Regression and OnBoarding tags, and not with skip, Manual or wip tags. Refer this post to learn more about Meta Filtering with JBehave

Another example of maven parameters -

mvn clean integration-test 
-Duse.sso.login=true 
-Dcontext.view.show=false 
-Dusers.dir="c:\jenkins\jbehave" 
-Dbrowser=firefox -Denv=test1 
-Dmeta.filters="${PARM_META_FILTERS} -broken -wip -skip -manual"


We can declare parameter names like ${suiteXmlLFile} in the POM and then specify that in the mvn command to pass the value at run time.
<suiteXmlFile>${suiteXmlFile}</suiteXmlFile>

mvn test -DsuiteXmlFile=./Config/parallelTestNG.xml

Maven commands are case sensitive hence, use exact names while specifying and reading values.


One cool trick is that you can validate the value that was passed via maven correctly or not, by looking up that property [browserType] in the 'System Properties' pane of JVisualVM for your IDE - intelliJ or Eclipse.